Examples with AddElements uk.gov.gchq.gaffer.operation.impl.add.AddElements used on opensource projects

Example 31 with AddElements

use of uk.gov.gchq.gaffer.operation.impl.add.AddElements in project gaffer-doc by gchq.

the class MultipleEdges method run.

@Override
public CloseableIterable<? extends Element> run() throws OperationException, IOException {
    // [generate] Create some edges from the simple data file using our Road Use generator class
    // ---------------------------------------------------------
    final List<Element> elements = new ArrayList<>();
    final RoadAndRoadUseElementGenerator dataGenerator = new RoadAndRoadUseElementGenerator();
    for (final String line : IOUtils.readLines(StreamUtil.openStream(getClass(), dataPath))) {
        Iterables.addAll(elements, dataGenerator._apply(line));
    }
    // ---------------------------------------------------------
    print("Elements generated from the data file.");
    for (final Element element : elements) {
        print("GENERATED_EDGES", element.toString());
    }
    print("");
    // [graph] Create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Add the edges to the graph
    // ---------------------------------------------------------
    final AddElements addElements = new AddElements.Builder().input(elements).build();
    graph.execute(addElements, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [get simple] Get all the edges related to vertex 10
    // ---------------------------------------------------------
    final GetElements getElements = new GetElements.Builder().input(new EntitySeed("10")).build();
    final CloseableIterable<? extends Element> results = graph.execute(getElements, user);
    // ---------------------------------------------------------
    print("\nAll elements containing vertex 10");
    print("\nNotice that the edges are aggregated within their groups");
    for (final Element e : results) {
        print("GET_ELEMENTS_RESULT", e.toString());
    }
    // [get] Rerun the previous query with a View to specify which subset of results we want
    // ---------------------------------------------------------
    final View view = new View.Builder().edge("RoadHasJunction").build();
    final GetElements getRoadHasJunctionEdges = new GetElements.Builder().input(new EntitySeed("10")).view(view).build();
    final CloseableIterable<? extends Element> filteredResults = graph.execute(getRoadHasJunctionEdges, user);
    // ---------------------------------------------------------
    print("\nAll RoadHasJunction edges containing vertex 10\n");
    for (final Element e : filteredResults) {
        print("GET_ROAD_HAS_JUNCTION_EDGES_RESULT", e.toString());
    }
    return filteredResults;
}

Example 32 with AddElements

use of uk.gov.gchq.gaffer.operation.impl.add.AddElements in project gaffer-doc by gchq.

the class Subgraphs method run.

@Override
public Iterable<? extends Element> run() throws OperationException, IOException {
    // / [graph] create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Create a data generator and add the edges to the graph using an operation chain consisting of:
    // generateElements - generating edges from the data (note these are directed edges)
    // addElements - add the edges to the graph
    // ---------------------------------------------------------
    final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<String>().generator(new RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator()).input(IOUtils.readLines(StreamUtil.openStream(getClass(), "RoadAndRoadUseWithTimesAndCardinalities/data.txt"))).build()).then(new AddElements()).build();
    graph.execute(addOpChain, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [get] Create a sub graph
    // Start getting related edges with the given seeds.
    // Then update the export with the results
    // Between each hop we need to extract the destination vertices of the
    // previous edges and convert them to EntitySeeds.
    // Finally finish off by returning all the edges in the export.
    // ---------------------------------------------------------
    final OperationChain<Iterable<?>> opChain = new OperationChain.Builder().first(new GetElements.Builder().input(new EntitySeed("M5")).inOutType(IncludeIncomingOutgoingType.OUTGOING).view(new View.Builder().edge("RoadHasJunction").build()).build()).then(new ExportToSet<>()).then(new ToVertices.Builder().edgeVertices(ToVertices.EdgeVertices.DESTINATION).build()).then(new ToEntitySeeds()).then(new GetElements.Builder().inOutType(IncludeIncomingOutgoingType.OUTGOING).view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().preAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(1L)).build()).build()).build()).build()).then(new ExportToSet<>()).then(new DiscardOutput()).then(new GetSetExport()).build();
    // ---------------------------------------------------------
    final Iterable<? extends Element> subGraph = (Iterable<? extends Element>) graph.execute(opChain, user);
    print("\nSub graph:");
    for (final Element edge : subGraph) {
        print("SUB_GRAPH", edge.toString());
    }
    return subGraph;
}

Example 33 with AddElements

use of uk.gov.gchq.gaffer.operation.impl.add.AddElements in project gaffer-doc by gchq.

the class Transforms method run.

@Override
public CloseableIterable<? extends Element> run() throws OperationException, IOException {
    // [generate] Create some edges from the simple data file using our Road Use generator class
    // ---------------------------------------------------------
    final List<Element> elements = new ArrayList<>();
    final RoadAndRoadUseElementGenerator dataGenerator = new RoadAndRoadUseElementGenerator();
    for (final String line : IOUtils.readLines(StreamUtil.openStream(getClass(), dataPath))) {
        Iterables.addAll(elements, dataGenerator._apply(line));
    }
    // ---------------------------------------------------------
    print("Elements generated from the data file.");
    for (final Element element : elements) {
        print("GENERATED_EDGES", element.toString());
    }
    print("");
    // [graph] Create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Add the edges to the graph
    // ---------------------------------------------------------
    final AddElements addElements = new AddElements.Builder().input(elements).build();
    graph.execute(addElements, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [get simple] get all the edges that contain the vertex "1"
    // ---------------------------------------------------------
    final GetElements getEdges = new GetElements.Builder().input(new EntitySeed("10")).build();
    final CloseableIterable<? extends Element> results = graph.execute(getEdges, user);
    // ---------------------------------------------------------
    print("\nAll edges containing the vertex 10. The counts and 'things' have been aggregated\n");
    for (final Element e : results) {
        print("GET_ELEMENTS_RESULT", e.toString());
    }
    // [transform] Create a description transient property using an element transformer
    // ---------------------------------------------------------
    final ElementTransformer descriptionTransformer = new ElementTransformer.Builder().select("SOURCE", "DESTINATION", "count").execute(new DescriptionTransform()).project("description").build();
    // ---------------------------------------------------------
    printJson("TRANSFORM", descriptionTransformer);
    // [get] Add the element transformer to the view and run the query
    // ---------------------------------------------------------
    final GetElements getEdgesWithDescription = new GetElements.Builder().input(new EntitySeed("10")).view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().transientProperty("description", String.class).transformer(descriptionTransformer).build()).build()).build();
    final CloseableIterable<? extends Element> resultsWithDescription = graph.execute(getEdgesWithDescription, user);
    // ---------------------------------------------------------
    printJsonAndPython("GET", getEdgesWithDescription);
    print("\nWe can add a new property to the edges that is calculated from the aggregated values of other properties\n");
    for (final Element e : resultsWithDescription) {
        print("GET_ELEMENTS_WITH_DESCRIPTION_RESULT", e.toString());
    }
    // [get with no count]
    // ---------------------------------------------------------
    final GetElements getEdgesWithDescriptionAndNoCount = new GetElements.Builder().input(new EntitySeed("10")).view(new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().transientProperty("description", String.class).transformer(descriptionTransformer).excludeProperties("count").build()).build()).build();
    final CloseableIterable<? extends Element> resultsWithDescriptionAndNoCount = graph.execute(getEdgesWithDescriptionAndNoCount, user);
    // ---------------------------------------------------------
    printJsonAndPython("GET_WITH_NO_COUNT", getEdgesWithDescriptionAndNoCount);
    print("\nAnd the result without the count property:\n");
    for (final Element e : resultsWithDescriptionAndNoCount) {
        print("GET_ELEMENTS_WITH_DESCRIPTION_AND_NO_COUNT_RESULT", e.toString());
    }
    return resultsWithDescriptionAndNoCount;
}

Example 34 with AddElements

use of uk.gov.gchq.gaffer.operation.impl.add.AddElements in project gaffer-doc by gchq.

the class Views method run.

public CloseableIterable<? extends Element> run() throws OperationException, IOException {
    // [graph] create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] Create a data generator and add the edges to the graph using an operation chain consisting of:
    // generateElements - generating edges from the data (note these are directed edges)
    // addElements - add the edges to the graph
    // ---------------------------------------------------------
    final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<String>().generator(new RoadAndRoadUseWithTimesAndCardinalitiesElementGenerator()).input(IOUtils.readLines(StreamUtil.openStream(getClass(), "RoadAndRoadUseWithTimesAndCardinalities/data.txt"))).build()).then(new AddElements()).build();
    graph.execute(addOpChain, user);
    // ---------------------------------------------------------
    print("The elements have been added.");
    // [view with groups]
    // ---------------------------------------------------------
    final View viewWithGroups = new View.Builder().edge("RoadUse").edge("RoadHasJunction").entity("Cardinality").build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GROUPS", viewWithGroups);
    // [view with filters]
    // ---------------------------------------------------------
    final View viewWithFilters = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).build()).build()).entity("Cardinality", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(5L)).select("hllp").execute(new HyperLogLogPlusIsLessThan(10L)).build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_FILTERS", viewWithFilters);
    // [view with removed properties]
    // ---------------------------------------------------------
    final View viewWithRemovedProperties = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().properties("count").build()).entity("Cardinality", new ViewElementDefinition.Builder().excludeProperties("hllp", "edgeGroups").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_REMOVED_PROPERTIES", viewWithRemovedProperties);
    // [view with global filter] run query with a global filter to return only elements with a count more than 2
    // ---------------------------------------------------------
    final View viewWithGlobalFilter = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_FILTER", viewWithGlobalFilter);
    // [view with global and specific filters]
    // ---------------------------------------------------------
    final View globalAndSpecificView = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(0L)).build()).build()).edge("RoadUse", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).build()).build()).entity("Cardinality").build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_FILTERS", globalAndSpecificView);
    // [view with global and specific filters expanded]
    // ---------------------------------------------------------
    final View globalAndSpecificViewExpanded = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(2L)).select("count").execute(new IsMoreThan(0L)).build()).build()).entity("Cardinality", new ViewElementDefinition.Builder().postAggregationFilter(new ElementFilter.Builder().select("count").execute(new IsMoreThan(0L)).build()).build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_FILTERS_EXPANDED", globalAndSpecificViewExpanded);
    // [view with global removed properties]
    // ---------------------------------------------------------
    final View viewWithGlobalRemovedProperties = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().properties("count").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_REMOVED_PROPERTIES", viewWithGlobalRemovedProperties);
    // [view with global and specific removed properties]
    // ---------------------------------------------------------
    final View viewWithGlobalAndSpecificRemovedProperties = new View.Builder().globalElements(new GlobalViewElementDefinition.Builder().properties("count").build()).edge("RoadUse").entity("Cardinality", new ViewElementDefinition.Builder().properties("hllp").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_REMOVED_PROPERTIES", viewWithGlobalAndSpecificRemovedProperties);
    // [view with global and specific removed properties expanded]
    // ---------------------------------------------------------
    final View viewWithGlobalAndSpecificRemovedPropertiesExpanded = new View.Builder().edge("RoadUse", new ViewElementDefinition.Builder().properties("count").build()).entity("Cardinality", new ViewElementDefinition.Builder().properties("hllp").build()).build();
    // ---------------------------------------------------------
    printJsonAndPythonWithClass("VIEW_WITH_GLOBAL_AND_SPECIFIC_REMOVED_PROPERTIES_EXPANDED", viewWithGlobalAndSpecificRemovedPropertiesExpanded);
    return null;
}

Example 35 with AddElements

use of uk.gov.gchq.gaffer.operation.impl.add.AddElements in project gaffer-doc by gchq.

the class BoundedTimestampSetWalkthrough method run.

@Override
public CloseableIterable<? extends Element> run() throws OperationException {
    // / [graph] create a graph using our schema and store properties
    // ---------------------------------------------------------
    final Graph graph = new Graph.Builder().config(getDefaultGraphConfig()).addSchemas(StreamUtil.openStreams(getClass(), schemaPath)).storeProperties(getDefaultStoreProperties()).build();
    // ---------------------------------------------------------
    // [user] Create a user
    // ---------------------------------------------------------
    final User user = new User("user01");
    // ---------------------------------------------------------
    // [add] addElements - add the edges to the graph
    // ---------------------------------------------------------
    final Set<String> dummyData = Collections.singleton("");
    final OperationChain<Void> addOpChain = new OperationChain.Builder().first(new GenerateElements.Builder<String>().generator(new BoundedTimestampSetElementGenerator()).input(dummyData).build()).then(new AddElements()).build();
    graph.execute(addOpChain, user);
    // ---------------------------------------------------------
    print("Added an edge A-B 3 times, each time with a BoundedTimestampSet containing a random time in 2017.");
    // [get] Get all edges
    // ---------------------------------------------------------
    CloseableIterable<? extends Element> allEdges = graph.execute(new GetAllElements(), user);
    // ---------------------------------------------------------
    print("\nAll edges:");
    for (final Element edge : allEdges) {
        print("GET_ALL_EDGES_RESULT", edge.toString());
    }
    return null;
}