Search in sources :

Example 11 with TrainingDatasetFilter

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter in project hopsworks by logicalclocks.

the class TrainingDatasetControllerTest method testconvertToFilterEntities_leftLogicRightFilter.

@Test
public void testconvertToFilterEntities_leftLogicRightFilter() throws Exception {
    // fg.feature > 1 and fg.feature > 2
    // "fg.feature > 1" stores as filter logic and "fg.feature > 2" stores as filter
    TrainingDataset trainingDataset = new TrainingDataset();
    Feature f1 = new Feature("test_f", "fg0");
    FilterLogic head = new FilterLogic();
    head.setType(AND);
    FilterLogic left = new FilterLogic();
    Filter left_left = new Filter(f1, GREATER_THAN, "1");
    left.setLeftFilter(left_left);
    left.setType(SINGLE);
    head.setLeftLogic(left);
    Filter right = new Filter(f1, GREATER_THAN, "2");
    head.setRightFilter(right);
    List<TrainingDatasetFilter> actual = target.convertToFilterEntities(head, trainingDataset, "L");
    List<TrainingDatasetFilter> expected = new ArrayList<>();
    expected.add(createTrainingDatasetFilter(null, AND, "L"));
    expected.add(createTrainingDatasetFilter(createTrainingDatasetFilterCondition("test_f", GREATER_THAN, "2"), SINGLE, "L.R"));
    expected.add(createTrainingDatasetFilter(createTrainingDatasetFilterCondition("test_f", GREATER_THAN, "1"), SINGLE, "L.L"));
    Assert.assertEquals(expected.size(), actual.size());
    Assert.assertTrue(expected.containsAll(actual));
    Assert.assertTrue(actual.containsAll(expected));
}
Also used : TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) Filter(io.hops.hopsworks.common.featurestore.query.filter.Filter) TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) SqlFilterLogic(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.SqlFilterLogic) FilterLogic(io.hops.hopsworks.common.featurestore.query.filter.FilterLogic) ArrayList(java.util.ArrayList) Feature(io.hops.hopsworks.common.featurestore.query.Feature) Test(org.junit.Test)

Example 12 with TrainingDatasetFilter

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter in project hopsworks by logicalclocks.

the class TrainingDatasetControllerTest method testconvertToFilterEntities_featureComparison.

@Test
public void testconvertToFilterEntities_featureComparison() throws Exception {
    // fg.feature > fg.otherFeature and fg.feature > fg1.otherFeature
    TrainingDataset trainingDataset = new TrainingDataset();
    Feature f1 = new Feature("test_f", "fg0");
    FilterLogic head = new FilterLogic();
    head.setType(AND);
    FilterValue filterValueLeft = new FilterValue(0, "fg0", "test_f1");
    Filter left = new Filter(f1, GREATER_THAN, filterValueLeft);
    head.setLeftFilter(left);
    FilterValue filterValueRight = new FilterValue(1, "fg1", "test_f2");
    Filter right = new Filter(f1, GREATER_THAN, filterValueRight);
    head.setRightFilter(right);
    List<TrainingDatasetFilter> actual = target.convertToFilterEntities(head, trainingDataset, "L");
    List<TrainingDatasetFilter> expected = new ArrayList<>();
    expected.add(createTrainingDatasetFilter(null, AND, "L"));
    expected.add(createTrainingDatasetFilter(createTrainingDatasetFilterCondition("test_f", GREATER_THAN, "test_f1", null, 0), SINGLE, "L.L"));
    expected.add(createTrainingDatasetFilter(createTrainingDatasetFilterCondition("test_f", GREATER_THAN, "test_f2", null, 1), SINGLE, "L.R"));
    Assert.assertEquals(expected.size(), actual.size());
    Assert.assertTrue(expected.containsAll(actual));
    Assert.assertTrue(actual.containsAll(expected));
}
Also used : TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) Filter(io.hops.hopsworks.common.featurestore.query.filter.Filter) TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) SqlFilterLogic(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.SqlFilterLogic) FilterLogic(io.hops.hopsworks.common.featurestore.query.filter.FilterLogic) ArrayList(java.util.ArrayList) Feature(io.hops.hopsworks.common.featurestore.query.Feature) FilterValue(io.hops.hopsworks.common.featurestore.query.filter.FilterValue) Test(org.junit.Test)

Example 13 with TrainingDatasetFilter

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter in project hopsworks by logicalclocks.

the class TrainingDatasetController method makeTrainingDatasetFilter.

private TrainingDatasetFilter makeTrainingDatasetFilter(String path, TrainingDataset trainingDataset, Filter filter, SqlFilterLogic type) {
    TrainingDatasetFilter trainingDatasetFilter = new TrainingDatasetFilter(trainingDataset);
    TrainingDatasetFilterCondition condition = filter == null ? null : convertFilter(filter, trainingDatasetFilter);
    trainingDatasetFilter.setCondition(condition);
    trainingDatasetFilter.setPath(path);
    trainingDatasetFilter.setType(type);
    return trainingDatasetFilter;
}
Also used : TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) TrainingDatasetFilterCondition(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilterCondition)

Example 14 with TrainingDatasetFilter

use of io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter in project hopsworks by logicalclocks.

the class TrainingDatasetController method convertToFilterLogic.

/**
 * Reconstruct {@link io.hops.hopsworks.common.featurestore.query.filter.FilterLogic} from a list of
 * {@link io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter} entity
 * Logic:
 * 1. get head node
 * 2. if type is  single, return Filter
 *    else get left/right children and assign left/right filterLogic
 *
 * @param trainingDatasetFilters
 * @param features
 * @param headPath
 * @return filter logic
 * @throws FeaturestoreException
 */
FilterLogic convertToFilterLogic(Collection<TrainingDatasetFilter> trainingDatasetFilters, Map<String, Feature> features, String headPath) throws FeaturestoreException {
    if (trainingDatasetFilters.size() == 0) {
        return null;
    }
    FilterLogic filterLogic = new FilterLogic();
    TrainingDatasetFilter headNode = trainingDatasetFilters.stream().filter(filter -> filter.getPath().equals(headPath)).findFirst().orElseThrow(() -> new FeaturestoreException(RESTCodes.FeaturestoreErrorCode.COULD_NOT_GET_QUERY_FILTER, Level.WARNING));
    filterLogic.setType(headNode.getType());
    if (headNode.getType().equals(SqlFilterLogic.SINGLE)) {
        Filter filter = convertToFilter(headNode.getCondition(), features);
        filterLogic.setLeftFilter(filter);
    } else {
        List<TrainingDatasetFilter> leftChildren = trainingDatasetFilters.stream().filter(filter -> filter.getPath().startsWith(headPath + ".L")).collect(Collectors.toList());
        List<TrainingDatasetFilter> rightChildren = trainingDatasetFilters.stream().filter(filter -> filter.getPath().startsWith(headPath + ".R")).collect(Collectors.toList());
        if (!leftChildren.isEmpty()) {
            if (leftChildren.size() == 1) {
                filterLogic.setLeftFilter(convertToFilter(leftChildren.get(0).getCondition(), features));
            } else {
                filterLogic.setLeftLogic(convertToFilterLogic(leftChildren, features, headPath + ".L"));
            }
        }
        if (!rightChildren.isEmpty()) {
            if (rightChildren.size() == 1) {
                filterLogic.setRightFilter(convertToFilter(rightChildren.get(0).getCondition(), features));
            } else {
                filterLogic.setRightLogic(convertToFilterLogic(rightChildren, features, headPath + ".R"));
            }
        }
    }
    return filterLogic;
}
Also used : TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) FeaturegroupType(io.hops.hopsworks.persistence.entity.featurestore.featuregroup.FeaturegroupType) Date(java.util.Date) Feature(io.hops.hopsworks.common.featurestore.query.Feature) HopsfsTrainingDatasetController(io.hops.hopsworks.common.featurestore.trainingdatasets.hopsfs.HopsfsTrainingDatasetController) HopsFSProvenanceController(io.hops.hopsworks.common.provenance.core.HopsFSProvenanceController) Settings(io.hops.hopsworks.common.util.Settings) TransactionAttributeType(javax.ejb.TransactionAttributeType) Map(java.util.Map) FilterValue(io.hops.hopsworks.common.featurestore.query.filter.FilterValue) FeatureView(io.hops.hopsworks.persistence.entity.featurestore.featureview.FeatureView) TrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset) TrainingDatasetSplit(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.split.TrainingDatasetSplit) Utils(io.hops.hopsworks.common.hdfs.Utils) StatisticsConfig(io.hops.hopsworks.persistence.entity.featurestore.statistics.StatisticsConfig) JoinType(org.apache.calcite.sql.JoinType) Stateless(javax.ejb.Stateless) TransformationFunctionFacade(io.hops.hopsworks.common.featurestore.transformationFunction.TransformationFunctionFacade) HopsfsTrainingDatasetFacade(io.hops.hopsworks.common.featurestore.trainingdatasets.hopsfs.HopsfsTrainingDatasetFacade) Collection(java.util.Collection) TrainingDatasetFeatureDTO(io.hops.hopsworks.common.featurestore.feature.TrainingDatasetFeatureDTO) Featuregroup(io.hops.hopsworks.persistence.entity.featurestore.featuregroup.Featuregroup) RESTCodes(io.hops.hopsworks.restutils.RESTCodes) Join(io.hops.hopsworks.common.featurestore.query.join.Join) StatisticColumnController(io.hops.hopsworks.common.featurestore.statistics.columns.StatisticColumnController) FeaturestoreConnectorFacade(io.hops.hopsworks.common.featurestore.storageconnectors.FeaturestoreConnectorFacade) Collectors(java.util.stream.Collectors) QueryController(io.hops.hopsworks.common.featurestore.query.QueryController) SqlFilterLogic(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.SqlFilterLogic) TransformationFunction(io.hops.hopsworks.persistence.entity.featurestore.transformationFunction.TransformationFunction) List(java.util.List) FeaturestoreFacade(io.hops.hopsworks.common.featurestore.FeaturestoreFacade) ExternalTrainingDatasetController(io.hops.hopsworks.common.featurestore.trainingdatasets.external.ExternalTrainingDatasetController) FeaturestoreUtils(io.hops.hopsworks.common.featurestore.utils.FeaturestoreUtils) Optional(java.util.Optional) FeaturestoreConnector(io.hops.hopsworks.persistence.entity.featurestore.storageconnector.FeaturestoreConnector) DistributedFsService(io.hops.hopsworks.common.hdfs.DistributedFsService) InodeController(io.hops.hopsworks.common.hdfs.inode.InodeController) DistributedFileSystemOps(io.hops.hopsworks.common.hdfs.DistributedFileSystemOps) TrainingDatasetJoinCondition(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetJoinCondition) FeaturestoreConnectorType(io.hops.hopsworks.persistence.entity.featurestore.storageconnector.FeaturestoreConnectorType) HashMap(java.util.HashMap) FeaturestoreActivityMeta(io.hops.hopsworks.persistence.entity.featurestore.activity.FeaturestoreActivityMeta) Streams(com.logicalclocks.shaded.com.google.common.collect.Streams) OnlineFeaturestoreController(io.hops.hopsworks.common.featurestore.online.OnlineFeaturestoreController) Project(io.hops.hopsworks.persistence.entity.project.Project) TrainingDatasetFilterCondition(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilterCondition) ArrayList(java.util.ArrayList) Level(java.util.logging.Level) HopsfsTrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.hopsfs.HopsfsTrainingDataset) FeaturestoreException(io.hops.hopsworks.exceptions.FeaturestoreException) FeaturestoreActivityFacade(io.hops.hopsworks.common.featurestore.activity.FeaturestoreActivityFacade) PitJoinController(io.hops.hopsworks.common.featurestore.query.pit.PitJoinController) TransactionAttribute(javax.ejb.TransactionAttribute) HdfsUsersController(io.hops.hopsworks.common.hdfs.HdfsUsersController) OnlineFeaturegroupController(io.hops.hopsworks.common.featurestore.featuregroup.online.OnlineFeaturegroupController) Query(io.hops.hopsworks.common.featurestore.query.Query) Filter(io.hops.hopsworks.common.featurestore.query.filter.Filter) TrainingDatasetType(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetType) Inode(io.hops.hopsworks.persistence.entity.hdfs.inode.Inode) ProvenanceException(io.hops.hopsworks.exceptions.ProvenanceException) EJB(javax.ejb.EJB) TrainingDatasetFeature(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFeature) StatisticsController(io.hops.hopsworks.common.featurestore.statistics.StatisticsController) ExternalTrainingDataset(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.external.ExternalTrainingDataset) IOException(java.io.IOException) Featurestore(io.hops.hopsworks.persistence.entity.featurestore.Featurestore) FeaturegroupController(io.hops.hopsworks.common.featurestore.featuregroup.FeaturegroupController) ServiceException(io.hops.hopsworks.exceptions.ServiceException) TimeTravelFormat(io.hops.hopsworks.persistence.entity.featurestore.featuregroup.cached.TimeTravelFormat) TrainingDatasetJoin(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetJoin) QueryDTO(io.hops.hopsworks.common.featurestore.query.QueryDTO) StatisticColumn(io.hops.hopsworks.persistence.entity.featurestore.statistics.StatisticColumn) Dataset(io.hops.hopsworks.persistence.entity.dataset.Dataset) FilterLogic(io.hops.hopsworks.common.featurestore.query.filter.FilterLogic) Users(io.hops.hopsworks.persistence.entity.user.Users) Comparator(java.util.Comparator) FeaturegroupDTO(io.hops.hopsworks.common.featurestore.featuregroup.FeaturegroupDTO) TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) TrainingDatasetFilter(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter) Filter(io.hops.hopsworks.common.featurestore.query.filter.Filter) SqlFilterLogic(io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.SqlFilterLogic) FilterLogic(io.hops.hopsworks.common.featurestore.query.filter.FilterLogic) FeaturestoreException(io.hops.hopsworks.exceptions.FeaturestoreException)

Aggregations

TrainingDatasetFilter (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilter)14 Feature (io.hops.hopsworks.common.featurestore.query.Feature)12 Filter (io.hops.hopsworks.common.featurestore.query.filter.Filter)12 FilterLogic (io.hops.hopsworks.common.featurestore.query.filter.FilterLogic)12 SqlFilterLogic (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.SqlFilterLogic)12 ArrayList (java.util.ArrayList)12 TrainingDataset (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDataset)8 Test (org.junit.Test)7 Featuregroup (io.hops.hopsworks.persistence.entity.featurestore.featuregroup.Featuregroup)5 FilterValue (io.hops.hopsworks.common.featurestore.query.filter.FilterValue)4 TrainingDatasetFilterCondition (io.hops.hopsworks.persistence.entity.featurestore.trainingdataset.TrainingDatasetFilterCondition)2 Streams (com.logicalclocks.shaded.com.google.common.collect.Streams)1 FeaturestoreFacade (io.hops.hopsworks.common.featurestore.FeaturestoreFacade)1 FeaturestoreActivityFacade (io.hops.hopsworks.common.featurestore.activity.FeaturestoreActivityFacade)1 TrainingDatasetFeatureDTO (io.hops.hopsworks.common.featurestore.feature.TrainingDatasetFeatureDTO)1 FeaturegroupController (io.hops.hopsworks.common.featurestore.featuregroup.FeaturegroupController)1 FeaturegroupDTO (io.hops.hopsworks.common.featurestore.featuregroup.FeaturegroupDTO)1 OnlineFeaturegroupController (io.hops.hopsworks.common.featurestore.featuregroup.online.OnlineFeaturegroupController)1 OnlineFeaturestoreController (io.hops.hopsworks.common.featurestore.online.OnlineFeaturestoreController)1 Query (io.hops.hopsworks.common.featurestore.query.Query)1