Examples with Element uk.gov.gchq.gaffer.data.element.Element used on opensource projects

Example 51 with Element

use of uk.gov.gchq.gaffer.data.element.Element in project Gaffer by gchq.

the class GetJavaRDDOfAllElementsExample method getJavaRddOfAllElements.

public void getJavaRddOfAllElements(final JavaSparkContext sc, final Graph graph) throws OperationException {
    ROOT_LOGGER.setLevel(Level.INFO);
    // Avoid using getMethodNameAsSentence as it messes up the formatting of the "RDD" part
    log("#### get Java RDD of elements\n");
    printGraph();
    ROOT_LOGGER.setLevel(Level.OFF);
    final GetJavaRDDOfAllElements operation = new GetJavaRDDOfAllElements.Builder().javaSparkContext(sc).build();
    final JavaRDD<Element> rdd = graph.execute(operation, new User("user01"));
    final List<Element> elements = rdd.collect();
    ROOT_LOGGER.setLevel(Level.INFO);
    printJava("GetJavaRDDOfAllElements<ElementSeed> operation = new GetJavaRDDOfAllElements.Builder<>()\n" + "                .javaSparkContext(sc)\n" + "                .build();\n" + "JavaRDD<Element> rdd = graph.execute(operation, new User(\"user01\"));\n" + "List<Element> elements = rdd.collect();");
    log("The results are:");
    log("```");
    for (final Element e : elements) {
        log(e.toString());
    }
    log("```");
    ROOT_LOGGER.setLevel(Level.OFF);
}

Example 52 with Element

use of uk.gov.gchq.gaffer.data.element.Element in project Gaffer by gchq.

the class AccumuloStoreRelation method buildScan.

/**
     * Creates a <code>DataFrame</code> of all {@link Element}s from the specified groups with columns that are not
     * required filtered out.
     * <p>
     * Currently this does not push the projection down to the store (i.e. it should be implemented in an iterator,
     * not in the transform). Issue 320 refers to this.
     *
     * @param requiredColumns The columns to return.
     * @return An {@link RDD} of {@link Row}s containing the requested columns.
     */
@Override
public RDD<Row> buildScan(final String[] requiredColumns) {
    try {
        LOGGER.info("Building scan with required columns: {}", StringUtils.join(requiredColumns, ','));
        LOGGER.info("Building GetRDDOfAllElements with view set to groups {}", StringUtils.join(groups, ','));
        final GetRDDOfAllElements operation = new GetRDDOfAllElements(sqlContext.sparkContext());
        operation.setView(view);
        final RDD<Element> rdd = store.execute(operation, user);
        return rdd.map(new ConvertElementToRow(new LinkedHashSet<>(Arrays.asList(requiredColumns)), propertyNeedsConversion, converterByProperty), ClassTagConstants.ROW_CLASS_TAG);
    } catch (final OperationException e) {
        LOGGER.error("OperationException while executing operation {}", e);
        return null;
    }
}

Example 53 with Element

use of uk.gov.gchq.gaffer.data.element.Element in project Gaffer by gchq.

the class GetJavaRDDOfAllElementsHandler method doOperation.

private JavaRDD<Element> doOperation(final GetJavaRDDOfAllElements operation, final Context context, final AccumuloStore accumuloStore) throws OperationException {
    final JavaSparkContext sparkContext = operation.getJavaSparkContext();
    final Configuration conf = getConfiguration(operation);
    addIterators(accumuloStore, conf, context.getUser(), operation);
    final JavaPairRDD<Element, NullWritable> pairRDD = sparkContext.newAPIHadoopRDD(conf, ElementInputFormat.class, Element.class, NullWritable.class);
    return pairRDD.map(new FirstElement());
}

Example 54 with Element

use of uk.gov.gchq.gaffer.data.element.Element in project Gaffer by gchq.

the class OperationServiceIT method shouldReturnChunkedElements.

@Test
public void shouldReturnChunkedElements() throws IOException {
    // Given
    RestApiTestUtil.addElements(DEFAULT_ELEMENTS);
    // When
    final Response response = RestApiTestUtil.executeOperationChainChunked(new OperationChain<>(new GetAllElements<>()));
    // Then
    final List<Element> results = readChunkedElements(response);
    verifyElements(DEFAULT_ELEMENTS, results);
}

Example 55 with Element

use of uk.gov.gchq.gaffer.data.element.Element in project Gaffer by gchq.

the class OperationServiceIT method shouldReturnAllElements.

@Test
public void shouldReturnAllElements() throws IOException {
    // Given
    RestApiTestUtil.addElements(DEFAULT_ELEMENTS);
    // When
    final Response response = RestApiTestUtil.executeOperation(new GetAllElements<>());
    // Then
    final List<Element> results = response.readEntity(new GenericType<List<Element>>() {
    });
    verifyElements(DEFAULT_ELEMENTS, results);
}