Search in sources :

Example 6 with TrainingDataset

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset in project hopsworks by logicalclocks.

the class TrainingDatasetService method bulkPutTags.

@ApiOperation(value = "Create or update tags(bulk) for a training dataset", response = TagsDTO.class)
@PUT
@Path("/{trainingDatasetId}/tags")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_SCIENTIST, AllowedProjectRoles.DATA_OWNER })
@JWTRequired(acceptedTokens = { Audience.API, Audience.JOB }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
@ApiKeyRequired(acceptedScopes = { ApiScope.FEATURESTORE }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response bulkPutTags(@Context SecurityContext sc, @Context UriInfo uriInfo, @ApiParam(value = "Id of the training dataset", required = true) @PathParam("trainingDatasetId") Integer trainingDatasetId, TagsDTO tags) throws MetadataException, FeaturestoreException, SchematizedTagException, DatasetException {
    verifyIdProvided(trainingDatasetId);
    Users user = jWTHelper.getUserPrincipal(sc);
    TrainingDataset trainingDataset = trainingDatasetController.getTrainingDatasetById(featurestore, trainingDatasetId);
    AttachTagResult result;
    if (tags.getItems().size() == 0) {
        result = tagController.upsert(project, user, featurestore, trainingDataset, tags.getName(), tags.getValue());
    } else {
        Map<String, String> newTags = new HashMap<>();
        for (TagsDTO tag : tags.getItems()) {
            newTags.put(tag.getName(), tag.getValue());
        }
        result = tagController.upsert(project, user, featurestore, trainingDataset, newTags);
    }
    ResourceRequest resourceRequest = new ResourceRequest(ResourceRequest.Name.TAGS);
    TagsDTO dto = tagBuilder.build(uriInfo, resourceRequest, project, featurestore.getId(), ResourceRequest.Name.TRAININGDATASETS.name(), trainingDatasetId, result.getItems());
    UriBuilder builder = uriInfo.getAbsolutePathBuilder();
    if (result.isCreated()) {
        return Response.created(builder.build()).entity(dto).build();
    } else {
        return Response.ok(builder.build()).entity(dto).build();
    }
}
Also used : TagsDTO(io.hops.hopsworks.common.tags.TagsDTO) HashMap(java.util.HashMap) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) AttachTagResult(io.hops.hopsworks.common.tags.AttachTagResult) Users(io.hops.hopsworks.persistence.entity.user.Users) ResourceRequest(io.hops.hopsworks.common.api.ResourceRequest) UriBuilder(javax.ws.rs.core.UriBuilder) Path(javax.ws.rs.Path) DatasetPath(io.hops.hopsworks.common.dataset.util.DatasetPath) Consumes(javax.ws.rs.Consumes) Produces(javax.ws.rs.Produces) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) ApiOperation(io.swagger.annotations.ApiOperation) ApiKeyRequired(io.hops.hopsworks.api.filter.apiKey.ApiKeyRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles) PUT(javax.ws.rs.PUT)

Example 7 with TrainingDataset

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset in project hopsworks by logicalclocks.

the class TrainingDatasetService method compute.

@POST
@Path("/{trainingDatasetId}/compute")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
@ApiOperation(value = "Setup a job to compute and write a training dataset", response = JobDTO.class)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_OWNER, AllowedProjectRoles.DATA_SCIENTIST })
@JWTRequired(acceptedTokens = { Audience.API, Audience.JOB }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
@ApiKeyRequired(acceptedScopes = { ApiScope.DATASET_VIEW, ApiScope.FEATURESTORE }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response compute(@Context UriInfo uriInfo, @Context SecurityContext sc, @PathParam("trainingDatasetId") Integer trainingDatasetId, TrainingDatasetJobConf trainingDatasetJobConf) throws FeaturestoreException, ServiceException, JobException, ProjectException, GenericException {
    verifyIdProvided(trainingDatasetId);
    Users user = jWTHelper.getUserPrincipal(sc);
    TrainingDataset trainingDataset = trainingDatasetController.getTrainingDatasetById(featurestore, trainingDatasetId);
    Map<String, String> writeOptions = null;
    if (trainingDatasetJobConf.getWriteOptions() != null) {
        writeOptions = trainingDatasetJobConf.getWriteOptions().stream().collect(Collectors.toMap(OptionDTO::getName, OptionDTO::getValue));
    }
    Jobs job = fsJobManagerController.setupTrainingDatasetJob(project, user, trainingDataset, trainingDatasetJobConf.getQuery(), trainingDatasetJobConf.getOverwrite(), writeOptions, trainingDatasetJobConf.getSparkJobConfiguration());
    JobDTO jobDTO = jobsBuilder.build(uriInfo, new ResourceRequest(ResourceRequest.Name.JOBS), job);
    return Response.created(jobDTO.getHref()).entity(jobDTO).build();
}
Also used : JobDTO(io.hops.hopsworks.api.jobs.JobDTO) Jobs(io.hops.hopsworks.persistence.entity.jobs.description.Jobs) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) OptionDTO(io.hops.hopsworks.common.featurestore.OptionDTO) Users(io.hops.hopsworks.persistence.entity.user.Users) ResourceRequest(io.hops.hopsworks.common.api.ResourceRequest) Path(javax.ws.rs.Path) DatasetPath(io.hops.hopsworks.common.dataset.util.DatasetPath) POST(javax.ws.rs.POST) Consumes(javax.ws.rs.Consumes) Produces(javax.ws.rs.Produces) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) ApiOperation(io.swagger.annotations.ApiOperation) ApiKeyRequired(io.hops.hopsworks.api.filter.apiKey.ApiKeyRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Example 8 with TrainingDataset

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset in project hopsworks by logicalclocks.

the class TrainingDatasetService method getTag.

@ApiOperation(value = "Get tag attached to a training dataset", response = TagsDTO.class)
@GET
@Path("/{trainingDatasetId}/tags/{name}")
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_SCIENTIST, AllowedProjectRoles.DATA_OWNER })
@JWTRequired(acceptedTokens = { Audience.API, Audience.JOB }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
@ApiKeyRequired(acceptedScopes = { ApiScope.FEATURESTORE }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response getTag(@Context SecurityContext sc, @Context UriInfo uriInfo, @ApiParam(value = "Id of the training dataset", required = true) @PathParam("trainingDatasetId") Integer trainingDatasetId, @ApiParam(value = "Name of the tag", required = true) @PathParam("name") String name, @BeanParam TagsExpansionBeanParam tagsExpansionBeanParam) throws DatasetException, MetadataException, FeaturestoreException, SchematizedTagException {
    verifyIdProvided(trainingDatasetId);
    Users user = jWTHelper.getUserPrincipal(sc);
    TrainingDataset trainingDataset = trainingDatasetController.getTrainingDatasetById(featurestore, trainingDatasetId);
    Map<String, String> result = new HashMap<>();
    result.put(name, tagController.get(project, user, featurestore, trainingDataset, name));
    ResourceRequest resourceRequest = new ResourceRequest(ResourceRequest.Name.TAGS);
    resourceRequest.setExpansions(tagsExpansionBeanParam.getResources());
    TagsDTO dto = tagBuilder.build(uriInfo, resourceRequest, project, featurestore.getId(), ResourceRequest.Name.TRAININGDATASETS.name(), trainingDatasetId, result);
    return Response.status(Response.Status.OK).entity(dto).build();
}
Also used : TagsDTO(io.hops.hopsworks.common.tags.TagsDTO) HashMap(java.util.HashMap) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) Users(io.hops.hopsworks.persistence.entity.user.Users) ResourceRequest(io.hops.hopsworks.common.api.ResourceRequest) Path(javax.ws.rs.Path) DatasetPath(io.hops.hopsworks.common.dataset.util.DatasetPath) Produces(javax.ws.rs.Produces) GET(javax.ws.rs.GET) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) ApiOperation(io.swagger.annotations.ApiOperation) ApiKeyRequired(io.hops.hopsworks.api.filter.apiKey.ApiKeyRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Example 9 with TrainingDataset

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset in project hopsworks by logicalclocks.

the class TrainingDatasetService method deleteTags.

@ApiOperation(value = "Delete all attached tags to training dataset")
@DELETE
@Path("/{trainingDatasetId}/tags")
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_SCIENTIST, AllowedProjectRoles.DATA_OWNER })
@JWTRequired(acceptedTokens = { Audience.API, Audience.JOB }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
@ApiKeyRequired(acceptedScopes = { ApiScope.FEATURESTORE }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response deleteTags(@Context SecurityContext sc, @ApiParam(value = "Id of the training dataset", required = true) @PathParam("trainingDatasetId") Integer trainingDatasetId) throws DatasetException, MetadataException, SchematizedTagException, FeaturestoreException {
    verifyIdProvided(trainingDatasetId);
    Users user = jWTHelper.getUserPrincipal(sc);
    TrainingDataset trainingDataset = trainingDatasetController.getTrainingDatasetById(featurestore, trainingDatasetId);
    tagController.deleteAll(project, user, featurestore, trainingDataset);
    return Response.noContent().build();
}
Also used : TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) Users(io.hops.hopsworks.persistence.entity.user.Users) Path(javax.ws.rs.Path) DatasetPath(io.hops.hopsworks.common.dataset.util.DatasetPath) DELETE(javax.ws.rs.DELETE) Produces(javax.ws.rs.Produces) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) ApiOperation(io.swagger.annotations.ApiOperation) ApiKeyRequired(io.hops.hopsworks.api.filter.apiKey.ApiKeyRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Example 10 with TrainingDataset

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset in project hopsworks by logicalclocks.

the class TrainingDatasetService method getTags.

@ApiOperation(value = "Get all tags attached to a training dataset", response = TagsDTO.class)
@GET
@Path("/{trainingDatasetId}/tags")
@Produces(MediaType.APPLICATION_JSON)
@AllowedProjectRoles({ AllowedProjectRoles.DATA_SCIENTIST, AllowedProjectRoles.DATA_OWNER })
@JWTRequired(acceptedTokens = { Audience.API, Audience.JOB }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
@ApiKeyRequired(acceptedScopes = { ApiScope.FEATURESTORE }, allowedUserRoles = { "HOPS_ADMIN", "HOPS_USER" })
public Response getTags(@Context SecurityContext sc, @Context UriInfo uriInfo, @ApiParam(value = "Id of the training dataset", required = true) @PathParam("trainingDatasetId") Integer trainingDatasetId, @BeanParam TagsExpansionBeanParam tagsExpansionBeanParam) throws DatasetException, MetadataException, FeaturestoreException, SchematizedTagException {
    verifyIdProvided(trainingDatasetId);
    Users user = jWTHelper.getUserPrincipal(sc);
    TrainingDataset trainingDataset = trainingDatasetController.getTrainingDatasetById(featurestore, trainingDatasetId);
    Map<String, String> result = tagController.getAll(project, user, featurestore, trainingDataset);
    ResourceRequest resourceRequest = new ResourceRequest(ResourceRequest.Name.TAGS);
    resourceRequest.setExpansions(tagsExpansionBeanParam.getResources());
    TagsDTO dto = tagBuilder.build(uriInfo, resourceRequest, project, featurestore.getId(), ResourceRequest.Name.TRAININGDATASETS.name(), trainingDatasetId, result);
    return Response.status(Response.Status.OK).entity(dto).build();
}
Also used : TagsDTO(io.hops.hopsworks.common.tags.TagsDTO) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) Users(io.hops.hopsworks.persistence.entity.user.Users) ResourceRequest(io.hops.hopsworks.common.api.ResourceRequest) Path(javax.ws.rs.Path) DatasetPath(io.hops.hopsworks.common.dataset.util.DatasetPath) Produces(javax.ws.rs.Produces) GET(javax.ws.rs.GET) JWTRequired(io.hops.hopsworks.jwt.annotation.JWTRequired) ApiOperation(io.swagger.annotations.ApiOperation) ApiKeyRequired(io.hops.hopsworks.api.filter.apiKey.ApiKeyRequired) AllowedProjectRoles(io.hops.hopsworks.api.filter.AllowedProjectRoles)

Aggregations

TrainingDataset (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset)29 Users (io.hops.hopsworks.persistence.entity.user.Users)12 ArrayList (java.util.ArrayList)11 Feature (io.hops.hopsworks.common.featurestore.query.Feature)10 Filter (io.hops.hopsworks.common.featurestore.query.filter.Filter)10 FilterLogic (io.hops.hopsworks.common.featurestore.query.filter.FilterLogic)10 Path (javax.ws.rs.Path)10 DatasetPath (io.hops.hopsworks.common.dataset.util.DatasetPath)9 JWTRequired (io.hops.hopsworks.jwt.annotation.JWTRequired)9 SqlFilterLogic (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.SqlFilterLogic)9 TrainingDatasetFilter (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter)9 ApiOperation (io.swagger.annotations.ApiOperation)9 AllowedProjectRoles (io.hops.hopsworks.api.filter.AllowedProjectRoles)8 ApiKeyRequired (io.hops.hopsworks.api.filter.apiKey.ApiKeyRequired)8 ResourceRequest (io.hops.hopsworks.common.api.ResourceRequest)8 FeaturestoreException (io.hops.hopsworks.exceptions.FeaturestoreException)8 ExternalTrainingDataset (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.external.ExternalTrainingDataset)8 HopsfsTrainingDataset (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.hopsfs.HopsfsTrainingDataset)8 Produces (javax.ws.rs.Produces)8 Test (org.junit.Test)8