Examples with WriteData uk.gov.gchq.gaffer.parquetstore.operation.handler.spark.utilities.WriteData used on opensource projects

Example 1 with WriteData

use of uk.gov.gchq.gaffer.parquetstore.operation.handler.spark.utilities.WriteData in project Gaffer by gchq.

the class AddElementsFromRDD method writeInputData.

/**
 * Writes the provided {@link JavaRDD} of {@link Element}s to files split by group and input partition (i.e. the
 * partition of the input {@link JavaRDD}, not the partition of the existing graph). The data is
 * written in the order the {@link JavaRDD} provides it, with no sorting or aggregation.
 *
 * @param input the JavaRDD of Elements
 */
private void writeInputData(final JavaRDD<Element> input) {
    LOGGER.info("Writing data for input RDD");
    final Function<String, String> groupToUnsortedUnaggregatedNewData = group -> getDirectory(group, false, false, false);
    input.foreachPartition(new WriteData(groupToUnsortedUnaggregatedNewData, schema, store.getProperties().getCompressionCodecName()));
}