Examples with FSDataOutputStream edu.iu.dsc.tws.api.data.FSDataOutputStream used on opensource projects

Example 1 with FSDataOutputStream

use of edu.iu.dsc.tws.api.data.FSDataOutputStream in project twister2 by DSC-SPIDAL.

the class HDFSFileStateStore method put.

@Override
public void put(String key, byte[] data) throws IOException {
    Path pathForKey = this.getPathForKey(key);
    FSDataOutputStream hadoopDataOutputStream = this.hdfs.create(pathForKey);
    IOUtils.copyBytes(new ByteArrayInputStream(data), hadoopDataOutputStream, data.length, true);
}

Example 2 with FSDataOutputStream

use of edu.iu.dsc.tws.api.data.FSDataOutputStream in project twister2 by DSC-SPIDAL.

the class DataGenerator method generateCSV.

/**
 * Generate a random csv file, we generate a csv with 10 attributes
 *
 * @param directory the path of the directory
 */
private static void generateCSV(Path directory, int numOfFiles, int sizeOfFile, int sizeMargin) throws IOException {
    FileSystem fs = FileSystemUtils.get(directory.toUri());
    Random random = new Random(System.currentTimeMillis());
    for (int i = 0; i < numOfFiles; i++) {
        FSDataOutputStream outputStream = fs.create(new Path(directory, generateRandom(10) + ".csv"));
        PrintWriter pw = new PrintWriter(outputStream);
        for (int j = 0; j < sizeOfFile + random.nextInt(sizeMargin); j++) {
            String row = generateCSVLine(10);
            pw.println(row);
        }
        pw.close();
    }
}

Example 3 with FSDataOutputStream

use of edu.iu.dsc.tws.api.data.FSDataOutputStream in project twister2 by DSC-SPIDAL.

the class DataGenerator method generateText.

private static void generateText(Path directory, int numOfFiles, int sizeOfFile, int sizeMargin) throws IOException {
    FileSystem fs = FileSystemUtils.get(directory.toUri());
    Random random = new Random(System.currentTimeMillis());
    for (int i = 0; i < numOfFiles; i++) {
        FSDataOutputStream outputStream = fs.create(new Path(directory, generateRandom(10) + ".txt"));
        PrintWriter pw = new PrintWriter(outputStream);
        for (int j = 0; j < sizeOfFile + random.nextInt(sizeMargin); j++) {
            String row = generateRandom(20 + random.nextInt(10));
            pw.println(row);
        }
        pw.close();
    }
}

Example 4 with FSDataOutputStream

use of edu.iu.dsc.tws.api.data.FSDataOutputStream in project twister2 by DSC-SPIDAL.

the class DataGenerator method generate.

public void generate(Path directory, int size, int dimension) {
    try {
        FileSystem fs = FileSystemUtils.get(directory.toUri(), config);
        if (fs.exists(directory)) {
            fs.delete(directory, true);
        }
        FSDataOutputStream outputStream = fs.create(new Path(directory, generateRandom(10) + ".txt"));
        PrintWriter pw = new PrintWriter(outputStream);
        String points = generatePoints(size, dimension, 100);
        pw.print(points);
        outputStream.sync();
        pw.close();
    } catch (IOException e) {
        throw new RuntimeException("Data Generation Error Occured", e);
    }
}

Example 5 with FSDataOutputStream

use of edu.iu.dsc.tws.api.data.FSDataOutputStream in project twister2 by DSC-SPIDAL.

the class FileOutputWriter method write.

public void write(int partition, T out) {
    FSDataOutputStream fsOut;
    if (!openStreams.containsKey(partition)) {
        Path path = new Path(outPath, "part-" + partition);
        try {
            fsOut = fs.create(path);
            // lets ask user to create its own output method
            createOutput(partition, fsOut);
            openStreams.put(partition, fsOut);
        } catch (IOException e) {
            throw new RuntimeException("Failed to create output stream for file: " + path, e);
        }
    }
    writeRecord(partition, out);
}