Examples with TextOutputWriter edu.iu.dsc.tws.data.api.out.TextOutputWriter used on opensource projects

Example 1 with TextOutputWriter

use of edu.iu.dsc.tws.data.api.out.TextOutputWriter in project twister2 by DSC-SPIDAL.

the class DataParallelTask method prepare.

@Override
public void prepare(Config cfg, TaskContext context) {
    super.prepare(cfg, context);
    String directory = cfg.getStringValue(Constants.ARGS_INPUT_DIRECTORY);
    ExecutionRuntime runtime = (ExecutionRuntime) config.get(ExecutorContext.TWISTER2_RUNTIME_OBJECT);
    String outDir = cfg.getStringValue(Constants.ARGS_OUTPUT_DIRECTORY);
    boolean shared = cfg.getBooleanValue(Constants.ARGS_SHARED_FILE_SYSTEM);
    if (!shared) {
        this.source = runtime.createInput(cfg, context, new LocalTextInputPartitioner(new Path(directory), context.getParallelism()));
    } else {
        this.source = runtime.createInput(cfg, context, new SharedTextInputPartitioner(new Path(directory)));
    }
    this.sink = new DataSink<String>(cfg, new TextOutputWriter(FileSystem.WriteMode.OVERWRITE, new Path(outDir)));
}

Example 2 with TextOutputWriter

use of edu.iu.dsc.tws.data.api.out.TextOutputWriter in project twister2 by DSC-SPIDAL.

the class DataFileSink method prepare.

@Override
public void prepare(Config cfg, TaskContext context) {
    super.prepare(cfg, context);
    String outDir = cfg.getStringValue(DataObjectConstants.OUTPUT_DIRECTORY);
    this.datasink = new DataSink<>(cfg, new TextOutputWriter(FileSystem.WriteMode.OVERWRITE, new Path(outDir)));
}

Example 3 with TextOutputWriter

use of edu.iu.dsc.tws.data.api.out.TextOutputWriter in project twister2 by DSC-SPIDAL.

the class KMeansDataGenerator method generateText.

private static void generateText(Path directory, int numOfFiles, int sizeOfFile, int sizeMargin, int dimension, Config config) {
    for (int i = 0; i < numOfFiles; i++) {
        String points = generatePoints(sizeOfFile, dimension, sizeMargin);
        TextOutputWriter textOutputWriter = new TextOutputWriter(FileSystem.WriteMode.OVERWRITE, directory, config);
        textOutputWriter.createOutput();
        textOutputWriter.writeRecord(points);
        textOutputWriter.close();
    }
}