Search in sources :

Example 1 with FreqMapExtractor

use of uk.gov.gchq.gaffer.types.function.FreqMapExtractor in project gaffer-doc by gchq.

the class FreqMapExtractorExample method multiplyAllMapValuesBy10.

public void multiplyAllMapValuesBy10() {
    // ---------------------------------------------------------
    final FreqMapExtractor function = new FreqMapExtractor("key1");
    // ---------------------------------------------------------
    final FreqMap map1 = new FreqMap();
    map1.put("key1", 1L);
    map1.put("key2", 2L);
    map1.put("key3", 3L);
    final FreqMap map2 = new FreqMap();
    map2.put("key2", 2L);
    map2.put("key3", 3L);
    final Map<String, Long> map3 = new HashMap<>();
    map3.put("key1", 1L);
    map3.put("key2", 2L);
    map3.put("key3", 3L);
    runExample(function, null, map1, map2, map3, null);
}
Also used : FreqMap(uk.gov.gchq.gaffer.types.FreqMap) HashMap(java.util.HashMap) FreqMapExtractor(uk.gov.gchq.gaffer.types.function.FreqMapExtractor)

Example 2 with FreqMapExtractor

use of uk.gov.gchq.gaffer.types.function.FreqMapExtractor in project gaffer-doc by gchq.

the class FullExample method run.

@Override
public Iterable<? extends String> run() throws OperationException, IOException {
    // [graph] Create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    try (final CSVParser parser = CSVParser.parse(getClass().getResource("/FullExample/data.txt"), Charset.defaultCharset(), CSVFormat.DEFAULT.withFirstRecordAsHeader())) {
        final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<CSVRecord>().generator(new RoadTrafficCsvElementGenerator()).input(parser).build()).then(new AddElements()).build();
        graph.execute(addOpChain, user);
    }
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [get] Get all road junctions in the South West that were heavily used by buses in the year 2000.
    // ---------------------------------------------------------
    final OperationChain<Iterable<? extends String>> opChain = new OperationChain.Builder().first(new GetAdjacentIds.Builder().input(new EntitySeed("South West")).view(new View.Builder().edge("RegionContainsLocation").build()).build()).then(new GetAdjacentIds.Builder().view(new View.Builder().edge("LocationContainsRoad").build()).build()).then(new ToSet<>()).then(new GetAdjacentIds.Builder().view(new View.Builder().edge("RoadHasJunction").build()).build()).then(new GetElements.Builder().view(new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().groupBy().build()).entity("JunctionUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("startDate", "endDate").execute(new InDateRangeDual.Builder().start("2000/01/01").end("2001/01/01").build()).build()).postAggregationFilter(new ElementFilter.Builder().select("countByVehicleType").execute(new PredicateMap<>("BUS", new IsMoreThan(1000L))).build()).transientProperty("busCount", Long.class).transformer(new ElementTransformer.Builder().select("countByVehicleType").execute(new FreqMapExtractor("BUS")).project("busCount").build()).build()).build()).inOutType(SeededGraphFilters.IncludeIncomingOutgoingType.OUTGOING).build()).then(new Sort.Builder().comparators(new ElementPropertyComparator.Builder().groups("JunctionUse").property("busCount").reverse(true).build()).resultLimit(2).deduplicate(true).build()).then(new ToCsv.Builder().generator(new CsvGenerator.Builder().vertex("Junction").property("busCount", "Bus Count").build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPython("GET", opChain);
    final Iterable<? extends String> results = graph.execute(opChain, user);
    print("\nAll road junctions in the South West that were heavily used by buses in year 2000.");
    for (final String result : results) {
        print("RESULT", result);
    }
    return results;
}
Also used : AddElements(uk.gov.gchq.gaffer.operation.impl.add.AddElements) User(uk.gov.gchq.gaffer.user.User) GetAdjacentIds(uk.gov.gchq.gaffer.operation.impl.get.GetAdjacentIds) GetElements(uk.gov.gchq.gaffer.operation.impl.get.GetElements) ToSet(uk.gov.gchq.gaffer.operation.impl.output.ToSet) Sort(uk.gov.gchq.gaffer.operation.impl.compare.Sort) RoadTrafficCsvElementGenerator(uk.gov.gchq.gaffer.traffic.generator.RoadTrafficCsvElementGenerator) ElementTransformer(uk.gov.gchq.gaffer.data.element.function.ElementTransformer) FreqMapExtractor(uk.gov.gchq.gaffer.types.function.FreqMapExtractor) CsvGenerator(uk.gov.gchq.gaffer.data.generator.CsvGenerator) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) ElementPropertyComparator(uk.gov.gchq.gaffer.data.element.comparison.ElementPropertyComparator) InDateRangeDual(uk.gov.gchq.koryphe.impl.predicate.range.InDateRangeDual) Graph(uk.gov.gchq.gaffer.graph.Graph) CSVParser(org.apache.commons.csv.CSVParser) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) EntitySeed(uk.gov.gchq.gaffer.operation.data.EntitySeed) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)

Example 3 with FreqMapExtractor

use of uk.gov.gchq.gaffer.types.function.FreqMapExtractor in project Gaffer by gchq.

the class Queries method runFullExample.

private void runFullExample(final Graph graph, final User user) throws OperationException {
    final OperationChain<Iterable<? extends String>> opChain = new OperationChain.Builder().first(new GetAdjacentIds.Builder().input(new EntitySeed("South West")).view(new View.Builder().edge("RegionContainsLocation").build()).build()).then(new GetAdjacentIds.Builder().view(new View.Builder().edge("LocationContainsRoad").build()).build()).then(new ToSet<>()).then(new GetAdjacentIds.Builder().view(new View.Builder().edge("RoadHasJunction").build()).build()).then(new GetElements.Builder().view(new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().groupBy().build()).entity("JunctionUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("startDate", "endDate").execute(new InDateRangeDual.Builder().start("2000/01/01").end("2001/01/01").build()).build()).postAggregationFilter(new ElementFilter.Builder().select("countByVehicleType").execute(new PredicateMap<>("BUS", new IsMoreThan(1000L))).build()).transientProperty("busCount", Long.class).transformer(new ElementTransformer.Builder().select("countByVehicleType").execute(new FreqMapExtractor("BUS")).project("busCount").build()).build()).build()).inOutType(SeededGraphFilters.IncludeIncomingOutgoingType.OUTGOING).build()).then(new Sort.Builder().comparators(new ElementPropertyComparator.Builder().groups("JunctionUse").property("busCount").reverse(true).build()).resultLimit(2).deduplicate(true).build()).then(new ToCsv.Builder().generator(new CsvGenerator.Builder().vertex("Junction").property("busCount", "Bus Count").build()).build()).build();
    final Iterable<? extends String> results = graph.execute(opChain, user);
    System.out.println("Full example results:");
    for (final String result : results) {
        System.out.println(result);
    }
}
Also used : GetAdjacentIds(uk.gov.gchq.gaffer.operation.impl.get.GetAdjacentIds) Builder(uk.gov.gchq.gaffer.graph.Graph.Builder) GetElements(uk.gov.gchq.gaffer.operation.impl.get.GetElements) ToSet(uk.gov.gchq.gaffer.operation.impl.output.ToSet) Sort(uk.gov.gchq.gaffer.operation.impl.compare.Sort) ElementTransformer(uk.gov.gchq.gaffer.data.element.function.ElementTransformer) FreqMapExtractor(uk.gov.gchq.gaffer.types.function.FreqMapExtractor) CsvGenerator(uk.gov.gchq.gaffer.data.generator.CsvGenerator) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) ElementPropertyComparator(uk.gov.gchq.gaffer.data.element.comparison.ElementPropertyComparator) InDateRangeDual(uk.gov.gchq.koryphe.impl.predicate.range.InDateRangeDual) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) EntitySeed(uk.gov.gchq.gaffer.operation.data.EntitySeed) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)

Example 4 with FreqMapExtractor

use of uk.gov.gchq.gaffer.types.function.FreqMapExtractor in project Gaffer by gchq.

the class RoadTrafficTestQueries method checkRoadJunctionsInSouthWestHeavilyUsedByBusesIn2000.

@Test
public void checkRoadJunctionsInSouthWestHeavilyUsedByBusesIn2000() throws OperationException, ParseException {
    assumeTrue(this.graph.hasTrait(StoreTrait.QUERY_AGGREGATION), "Skipping test as the store does not implement required trait.");
    assumeTrue(this.graph.hasTrait(StoreTrait.TRANSFORMATION), "Skipping test as the store does not implement required trait.");
    assumeTrue(this.graph.hasTrait(StoreTrait.PRE_AGGREGATION_FILTERING), "Skipping test as the store does not implement required trait.");
    assumeTrue(this.graph.hasTrait(StoreTrait.POST_AGGREGATION_FILTERING), "Skipping test as the store does not implement required trait.");
    assertNotNull(this.graph, "graph is null");
    final Date JAN_01_2000 = new SimpleDateFormat("yyyy-MM-dd").parse("2000-01-01");
    final Date JAN_01_2001 = new SimpleDateFormat("yyyy-MM-dd").parse("2001-01-01");
    final OperationChain<Iterable<? extends String>> opChain = new OperationChain.Builder().first(new GetAdjacentIds.Builder().input(new EntitySeed("South West")).view(new View.Builder().edge("RegionContainsLocation").build()).build()).then(new GetAdjacentIds.Builder().view(new View.Builder().edge("LocationContainsRoad").build()).build()).then(new ToSet<>()).then(new GetAdjacentIds.Builder().view(new View.Builder().edge("RoadHasJunction").build()).build()).then(new GetElements.Builder().view(new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().groupBy().build()).entity("JunctionUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("startDate").execute(new IsMoreThan(JAN_01_2000, true)).select("endDate").execute(new IsLessThan(JAN_01_2001, false)).build()).postAggregationFilter(new ElementFilter.Builder().select("countByVehicleType").execute(new PredicateMap<>("BUS", new IsMoreThan(1000L))).build()).transientProperty("busCount", Long.class).transformer(new ElementTransformer.Builder().select("countByVehicleType").execute(new FreqMapExtractor("BUS")).project("busCount").build()).build()).build()).inOutType(SeededGraphFilters.IncludeIncomingOutgoingType.OUTGOING).build()).then(new ToCsv.Builder().generator(new CsvGenerator.Builder().vertex("Junction").property("busCount", "Bus Count").build()).build()).build();
    Set<String> resultSet = new HashSet<>();
    final Iterable<? extends String> results = this.graph.execute(opChain, this.user);
    for (final String r : results) {
        resultSet.add(r);
    }
    assertEquals(SW_ROAD_JUNCTIONS_WITH_HEAVY_BUS_USAGE_IN_2000, resultSet);
}
Also used : CloseableIterable(uk.gov.gchq.gaffer.commonutil.iterable.CloseableIterable) PredicateMap(uk.gov.gchq.koryphe.predicate.PredicateMap) GlobalViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.GlobalViewElementDefinition) ViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.ViewElementDefinition) GlobalViewElementDefinition(uk.gov.gchq.gaffer.data.elementdefinition.view.GlobalViewElementDefinition) IsLessThan(uk.gov.gchq.koryphe.impl.predicate.IsLessThan) HashSet(java.util.HashSet) FreqMapExtractor(uk.gov.gchq.gaffer.types.function.FreqMapExtractor) CsvGenerator(uk.gov.gchq.gaffer.data.generator.CsvGenerator) View(uk.gov.gchq.gaffer.data.elementdefinition.view.View) Date(java.util.Date) OperationChain(uk.gov.gchq.gaffer.operation.OperationChain) ElementFilter(uk.gov.gchq.gaffer.data.element.function.ElementFilter) EntitySeed(uk.gov.gchq.gaffer.operation.data.EntitySeed) IsMoreThan(uk.gov.gchq.koryphe.impl.predicate.IsMoreThan) SimpleDateFormat(java.text.SimpleDateFormat) Test(org.junit.jupiter.api.Test)

Aggregations

FreqMapExtractor (uk.gov.gchq.gaffer.types.function.FreqMapExtractor)4 ElementFilter (uk.gov.gchq.gaffer.data.element.function.ElementFilter)3 View (uk.gov.gchq.gaffer.data.elementdefinition.view.View)3 CsvGenerator (uk.gov.gchq.gaffer.data.generator.CsvGenerator)3 EntitySeed (uk.gov.gchq.gaffer.operation.data.EntitySeed)3 IsMoreThan (uk.gov.gchq.koryphe.impl.predicate.IsMoreThan)3 ElementPropertyComparator (uk.gov.gchq.gaffer.data.element.comparison.ElementPropertyComparator)2 ElementTransformer (uk.gov.gchq.gaffer.data.element.function.ElementTransformer)2 OperationChain (uk.gov.gchq.gaffer.operation.OperationChain)2 Sort (uk.gov.gchq.gaffer.operation.impl.compare.Sort)2 GetAdjacentIds (uk.gov.gchq.gaffer.operation.impl.get.GetAdjacentIds)2 GetElements (uk.gov.gchq.gaffer.operation.impl.get.GetElements)2 ToSet (uk.gov.gchq.gaffer.operation.impl.output.ToSet)2 InDateRangeDual (uk.gov.gchq.koryphe.impl.predicate.range.InDateRangeDual)2 SimpleDateFormat (java.text.SimpleDateFormat)1 Date (java.util.Date)1 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 CSVParser (org.apache.commons.csv.CSVParser)1 Test (org.junit.jupiter.api.Test)1