Search in sources :

Example 11 with IsMoreThan

use of uk.gov.gchq.koryphe.impl.predicate.IsMoreThan in project gaffer-doc by gchq.

the class PredicateMapExample method freqMapIsMoreThanOrEqualTo2.

public void freqMapIsMoreThanOrEqualTo2() {
    // ---------------------------------------------------------
    final PredicateMap function = new PredicateMap("key1", new IsMoreThan(2L, true));
    // ---------------------------------------------------------
    final FreqMap map1 = new FreqMap();
    map1.put("key1", 1L);
    final FreqMap map2 = new FreqMap();
    map2.put("key1", 2L);
    final FreqMap map3 = new FreqMap();
    map3.put("key1", 3L);
    final FreqMap map4 = new FreqMap();
    map4.put("key1", 3L);
    map4.put("key2", 0L);
    final FreqMap map5 = new FreqMap();
    map5.put("key2", 3L);
    runExample(function, null, map1, map2, map3, map4, map5);
}
Also used : FreqMap(uk.gov.gchq.gaffer.types.FreqMap) PredicateMap(uk.gov.gchq.koryphe.predicate.PredicateMap) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)

Example 12 with IsMoreThan

use of uk.gov.gchq.koryphe.impl.predicate.IsMoreThan in project gaffer-doc by gchq.

the class Aggregation method run.

@Override
public CloseableIterable<? extends Element> run() throws OperationException, IOException {
    // [graph] create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Create a data generator and add the edges to the graph using an operation chain consisting of:
    // generateElements - generating edges from the data (note these are directed edges)
    // addElements - add the edges to the graph
    // ---------------------------------------------------------
    final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<String>().generator(new RoadAndRoadUseWithTimesElementGenerator()).input(IOUtils.readLines(StreamUtil.openStream(getClass(), dataPath))).build()).then(new AddElements()).build();
    graph.execute(addOpChain, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [get] Get all RoadUse edges
    // ---------------------------------------------------------
    final GetAllElements getAllRoadUseEdges = new GetAllElements.Builder().view(new View.Builder().edge("RoadUse").build()).build();
    final CloseableIterable<? extends Element> roadUseElements = graph.execute(getAllRoadUseEdges, user);
    // ---------------------------------------------------------
    print("\nAll RoadUse edges in daily time buckets:");
    for (final Element element : roadUseElements) {
        print("GET_ALL_EDGES_RESULT", element.toString());
    }
    // [get all edges summarised] Get all edges summarised (merge all time windows together)
    // This is achieved by overriding the 'groupBy' start and end time properties.
    // ---------------------------------------------------------
    final GetAllElements edgesSummarisedOperation = new GetAllElements.Builder().view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().groupBy().build()).build()).build();
    final CloseableIterable<? extends Element> edgesSummarised = graph.execute(edgesSummarisedOperation, user);
    // ---------------------------------------------------------
    print("\nAll edges summarised:");
    for (final Element edge : edgesSummarised) {
        print("GET_ALL_EDGES_SUMMARISED_RESULT", edge.toString());
    }
    // [get all edges summarised in time window] Get all edges summarised over a provided 2 day time period
    // This is achieved by overriding the 'groupBy' start and end time properties
    // and providing a filter.
    // ---------------------------------------------------------
    final GetAllElements edgesSummarisedInTimeWindowOperation = new GetAllElements.Builder().view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("startDate").execute(new IsMoreThan(MAY_01_2000, true)).select("endDate").execute(new IsLessThan(MAY_02_2000, false)).build()).groupBy().build()).build()).build();
    final CloseableIterable<? extends Element> edgesSummarisedInTimeWindow = graph.execute(edgesSummarisedInTimeWindowOperation, user);
    // ---------------------------------------------------------
    print("\nEdges in 2 day time window:");
    for (final Element edge : edgesSummarisedInTimeWindow) {
        print("GET_ALL_EDGES_SUMMARISED_IN_TIME_WINDOW_RESULT", edge.toString());
    }
    // [get all edges summarised in time window with min count] Now a bit more advanced.
    // At query time you can actually override the logic for how Gaffer
    // aggregates properties together. So by default the count property
    // is aggregated with Sum. At query time we could change that, so the
    // count property was aggregated with Max.
    // ---------------------------------------------------------
    final GetAllElements edgesSummarisedInTimeWindowWithMinCountOperation = new GetAllElements.Builder().view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("startDate").execute(new IsMoreThan(MAY_01_2000, true)).select("endDate").execute(new IsLessThan(MAY_03_2000, false)).build()).groupBy().aggregator(new ElementAggregator.Builder().select("count").execute(new Min()).build()).build()).build()).build();
    printJson("GET_ALL_EDGES_SUMMARISED_IN_TIME_WINDOW_RESULT_WITH_MIN_COUNT", edgesSummarisedInTimeWindowWithMinCountOperation);
    final CloseableIterable<? extends Element> edgesSummarisedInTimeWindowWithMinCount = graph.execute(edgesSummarisedInTimeWindowWithMinCountOperation, user);
    // ---------------------------------------------------------
    print("\nEdges in 3 day time window with min count:");
    for (final Element edge : edgesSummarisedInTimeWindowWithMinCount) {
        print("GET_ALL_EDGES_SUMMARISED_IN_TIME_WINDOW_RESULT_WITH_MIN_COUNT", edge.toString());
    }
    return edgesSummarisedInTimeWindowWithMinCount;
}
Also used : AddElements(uk.gov.gchq.gaffer.operation.impl.add.AddElements) User(uk.gov.gchq.gaffer.user.User) RoadAndRoadUseWithTimesElementGenerator(uk.gov.gchq.gaffer.doc.user.generator.RoadAndRoadUseWithTimesElementGenerator) Element(uk.gov.gchq.gaffer.data.element.Element) ViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition) Graph(uk.gov.gchq.gaffer.graph.Graph) IsLessThan(uk.gov.gchq.koryphe.impl.predicate.IsLessThan) Min(uk.gov.gchq.koryphe.impl.binaryoperator.Min) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) GetAllElements(uk.gov.gchq.gaffer.operation.impl.get.GetAllElements) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan) ElementAggregator(uk.gov.gchq.gaffer.data.element.function.ElementAggregator)

Example 13 with IsMoreThan

use of uk.gov.gchq.koryphe.impl.predicate.IsMoreThan in project gaffer-doc by gchq.

the class Subgraphs method run.

@Override
public Iterable<? extends Element> run() throws OperationException, IOException {
    // / [graph] create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Create a data generator and add the edges to the graph using an operation chain consisting of:
    // generateElements - generating edges from the data (note these are directed edges)
    // addElements - add the edges to the graph
    // ---------------------------------------------------------
    final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<String>().generator(new RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator()).input(IOUtils.readLines(StreamUtil.openStream(getClass(), "RoadAndRoadUseWithTimesAndCardinalities/data.txt"))).build()).then(new AddElements()).build();
    graph.execute(addOpChain, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [get] Create a sub graph
    // Start getting related edges with the given seeds.
    // Then update the export with the results
    // Between each hop we need to extract the destination vertices of the
    // previous edges and convert them to EntitySeeds.
    // Finally finish off by returning all the edges in the export.
    // ---------------------------------------------------------
    final OperationChain<Iterable<?>> opChain = new OperationChain.Builder().first(new GetElements.Builder().input(new EntitySeed("M5")).inOutType(IncludeIncomingOutgoingType.OUTGOING).view(new View.Builder().edge("RoadHasJunction").build()).build()).then(new ExportToSet<>()).then(new ToVertices.Builder().edgeVertices(ToVertices.EdgeVertices.DESTINATION).build()).then(new ToEntitySeeds()).then(new GetElements.Builder().inOutType(IncludeIncomingOutgoingType.OUTGOING).view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(1L)).build()).build()).build()).build()).then(new ExportToSet<>()).then(new DiscardOutput()).then(new GetSetExport()).build();
    // ---------------------------------------------------------
    final Iterable<? extends Element> subGraph = (Iterable<? extends Element>) graph.execute(opChain, user);
    print("\nSub graph:");
    for (final Element edge : subGraph) {
        print("SUB_GRAPH", edge.toString());
    }
    return subGraph;
}
Also used : AddElements(uk.gov.gchq.gaffer.operation.impl.add.AddElements) User(uk.gov.gchq.gaffer.user.User) ExportToSet(uk.gov.gchq.gaffer.operation.impl.export.set.ExportToSet) ToEntitySeeds(uk.gov.gchq.gaffer.operation.impl.output.ToEntitySeeds) Element(uk.gov.gchq.gaffer.data.element.Element) GetElements(uk.gov.gchq.gaffer.operation.impl.get.GetElements) ViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition) GetSetExport(uk.gov.gchq.gaffer.operation.impl.export.set.GetSetExport) RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator(uk.gov.gchq.gaffer.doc.user.generator.RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator) ToVertices(uk.gov.gchq.gaffer.operation.impl.output.ToVertices) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) Graph(uk.gov.gchq.gaffer.graph.Graph) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) EntitySeed(uk.gov.gchq.gaffer.operation.data.EntitySeed) DiscardOutput(uk.gov.gchq.gaffer.operation.impl.DiscardOutput) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)

Example 14 with IsMoreThan

use of uk.gov.gchq.koryphe.impl.predicate.IsMoreThan in project gaffer-doc by gchq.

the class Views method run.

public CloseableIterable<? extends Element> run() throws OperationException, IOException {
    // [graph] create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Create a data generator and add the edges to the graph using an operation chain consisting of:
    // generateElements - generating edges from the data (note these are directed edges)
    // addElements - add the edges to the graph
    // ---------------------------------------------------------
    final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<String>().generator(new RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator()).input(IOUtils.readLines(StreamUtil.openStream(getClass(), "RoadAndRoadUseWithTimesAndCardinalities/data.txt"))).build()).then(new AddElements()).build();
    graph.execute(addOpChain, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [view with groups]
    // ---------------------------------------------------------
    final View viewWithGroups = new View.Builder().edge("RoadUse").edge("RoadHasJunction").entity("Cardinality").build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GROUPS", viewWithGroups);
    // [view with filters]
    // ---------------------------------------------------------
    final View viewWithFilters = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).build()).build()).entity("Cardinality", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(5L)).select("hllp").execute(new HyperLogLogPlusIsLessThan(10L)).build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_FILTERS", viewWithFilters);
    // [view with removed properties]
    // ---------------------------------------------------------
    final View viewWithRemovedProperties = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().properties("count").build()).entity("Cardinality", new ViewElementDefinition.Builder().excludeProperties("hllp", "edgeGroups").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_REMOVED_PROPERTIES", viewWithRemovedProperties);
    // [view with global filter] run query with a global filter to return only elements with a count more than 2
    // ---------------------------------------------------------
    final View viewWithGlobalFilter = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_FILTER", viewWithGlobalFilter);
    // [view with global and specific filters]
    // ---------------------------------------------------------
    final View globalAndSpecificView = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(0L)).build()).build()).edge("RoadUse", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).build()).build()).entity("Cardinality").build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_FILTERS", globalAndSpecificView);
    // [view with global and specific filters expanded]
    // ---------------------------------------------------------
    final View globalAndSpecificViewExpanded = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).select("count").execute(new IsMoreThan(0L)).build()).build()).entity("Cardinality", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(0L)).build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_FILTERS_EXPANDED", globalAndSpecificViewExpanded);
    // [view with global removed properties]
    // ---------------------------------------------------------
    final View viewWithGlobalRemovedProperties = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().properties("count").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_REMOVED_PROPERTIES", viewWithGlobalRemovedProperties);
    // [view with global and specific removed properties]
    // ---------------------------------------------------------
    final View viewWithGlobalAndSpecificRemovedProperties = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().properties("count").build()).edge("RoadUse").entity("Cardinality", new ViewElementDefinition.Builder().properties("hllp").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_REMOVED_PROPERTIES", viewWithGlobalAndSpecificRemovedProperties);
    // [view with global and specific removed properties expanded]
    // ---------------------------------------------------------
    final View viewWithGlobalAndSpecificRemovedPropertiesExpanded = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().properties("count").build()).entity("Cardinality", new ViewElementDefinition.Builder().properties("hllp").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_REMOVED_PROPERTIES_EXPANDED", viewWithGlobalAndSpecificRemovedPropertiesExpanded);
    return null;
}
Also used : AddElements(uk.gov.gchq.gaffer.operation.impl.add.AddElements) User(uk.gov.gchq.gaffer.user.User) GlobalViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.GlobalViewElementDefinition) ViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) Graph(uk.gov.gchq.gaffer.graph.Graph) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator(uk.gov.gchq.gaffer.doc.user.generator.RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) HyperLogLogPlusIsLessThan(uk.gov.gchq.gaffer.sketches.clearspring.cardinality.predicate.HyperLogLogPlusIsLessThan) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)

Example 15 with IsMoreThan

use of uk.gov.gchq.koryphe.impl.predicate.IsMoreThan in project Gaffer by gchq.

the class GetAllElementsHandlerTest method testGetAllElementsWithViewRestrictedByGroupAndAPreAggregationFilter.

@Test
public void testGetAllElementsWithViewRestrictedByGroupAndAPreAggregationFilter() throws OperationException {
    // Given
    final Graph graph = getGraph();
    final AddElements addElements = new AddElements.Builder().input(getElements()).build();
    graph.execute(addElements, new User());
    // When
    final GetAllElements getAllElements = new GetAllElements.Builder().view(new View.Builder().edge(BASIC_EDGE1, new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select(COUNT).execute(new IsMoreThan(5)).build()).build()).build()).build();
    final CloseableIterable<? extends Element> results = graph.execute(getAllElements, new User());
    // Then
    final Set<Element> resultsSet = new HashSet<>();
    Streams.toStream(results).forEach(resultsSet::add);
    final Set<Element> expectedResults = new HashSet<>();
    getElements().stream().filter(e -> e.getGroup().equals(BASIC_EDGE1) && ((int) e.getProperty(COUNT)) > 5).forEach(expectedResults::add);
    assertEquals(expectedResults, resultsSet);
}
Also used : AddElements(uk.gov.gchq.gaffer.operation.impl.add.AddElements) IntStream(java.util.stream.IntStream) StoreException(uk.gov.gchq.gaffer.store.StoreException) User(uk.gov.gchq.gaffer.user.User) HashMap(java.util.HashMap) GraphConfig(uk.gov.gchq.gaffer.graph.GraphConfig) Element(uk.gov.gchq.gaffer.data.element.Element) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) Graph(uk.gov.gchq.gaffer.graph.Graph) CloseableIterable(uk.gov.gchq.gaffer.commonutil.iterable.CloseableIterable) Map(java.util.Map) Edge(uk.gov.gchq.gaffer.data.element.Edge) Assertions.assertEquals(org.junit.jupiter.api.Assertions.assertEquals) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan) TestGroups(uk.gov.gchq.gaffer.commonutil.TestGroups) GetAllElements(uk.gov.gchq.gaffer.operation.impl.get.GetAllElements) StreamUtil(uk.gov.gchq.gaffer.commonutil.StreamUtil) DirectedType(uk.gov.gchq.gaffer.data.element.id.DirectedType) MapStoreProperties(uk.gov.gchq.gaffer.mapstore.MapStoreProperties) Set(java.util.Set) Entity(uk.gov.gchq.gaffer.data.element.Entity) Test(org.junit.jupiter.api.Test) List(java.util.List) Stream(java.util.stream.Stream) Schema(uk.gov.gchq.gaffer.store.schema.Schema) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) ViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition) AddElements(uk.gov.gchq.gaffer.operation.impl.add.AddElements) OperationException(uk.gov.gchq.gaffer.operation.OperationException) Streams(uk.gov.gchq.gaffer.commonutil.stream.Streams) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) User(uk.gov.gchq.gaffer.user.User) Element(uk.gov.gchq.gaffer.data.element.Element) Graph(uk.gov.gchq.gaffer.graph.Graph) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) GetAllElements(uk.gov.gchq.gaffer.operation.impl.get.GetAllElements) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan) HashSet(java.util.HashSet) Test(org.junit.jupiter.api.Test)

Aggregations

IsMoreThan (uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)72 Test (org.junit.jupiter.api.Test)42 ElementFilter (uk.gov.gchq.gaffer.data.element.function.ElementFilter)40 View (uk.gov.gchq.gaffer.data.elementdefinition.view.View)34 ViewElementDefinition (uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition)24 Element (uk.gov.gchq.gaffer.data.element.Element)22 EntitySeed (uk.gov.gchq.gaffer.operation.data.EntitySeed)22 Edge (uk.gov.gchq.gaffer.data.element.Edge)20 GetElements (uk.gov.gchq.gaffer.operation.impl.get.GetElements)20 User (uk.gov.gchq.gaffer.user.User)20 Graph (uk.gov.gchq.gaffer.graph.Graph)19 AddElements (uk.gov.gchq.gaffer.operation.impl.add.AddElements)19 ArrayList (java.util.ArrayList)18 HashSet (java.util.HashSet)16 Entity (uk.gov.gchq.gaffer.data.element.Entity)16 IsEqual (uk.gov.gchq.koryphe.impl.predicate.IsEqual)13 IsLessThan (uk.gov.gchq.koryphe.impl.predicate.IsLessThan)13 Set (java.util.Set)12 CloseableIterable (uk.gov.gchq.gaffer.commonutil.iterable.CloseableIterable)12 OperationException (uk.gov.gchq.gaffer.operation.OperationException)12