Search in sources :

Example 61 with Path

use of edu.iu.dsc.tws.api.data.Path in project twister2 by DSC-SPIDAL.

the class DataGenerator method generate.

public void generate(Path directory, int size, int dimension) {
    try {
        FileSystem fs = FileSystemUtils.get(directory.toUri(), config);
        if (fs.exists(directory)) {
            fs.delete(directory, true);
        }
        FSDataOutputStream outputStream = fs.create(new Path(directory, generateRandom(10) + ".txt"));
        PrintWriter pw = new PrintWriter(outputStream);
        String points = generatePoints(size, dimension, 100);
        pw.print(points);
        outputStream.sync();
        pw.close();
    } catch (IOException e) {
        throw new RuntimeException("Data Generation Error Occured", e);
    }
}
Also used : Path(edu.iu.dsc.tws.api.data.Path) FileSystem(edu.iu.dsc.tws.api.data.FileSystem) FSDataOutputStream(edu.iu.dsc.tws.api.data.FSDataOutputStream) IOException(java.io.IOException) PrintWriter(java.io.PrintWriter)

Aggregations

Path (edu.iu.dsc.tws.api.data.Path)61 IOException (java.io.IOException)23 FileSystem (edu.iu.dsc.tws.api.data.FileSystem)19 FileStatus (edu.iu.dsc.tws.api.data.FileStatus)14 ArrayList (java.util.ArrayList)12 Config (edu.iu.dsc.tws.api.config.Config)11 Twister2RuntimeException (edu.iu.dsc.tws.api.exceptions.Twister2RuntimeException)8 FileInputSplit (edu.iu.dsc.tws.data.api.splits.FileInputSplit)8 ExecutionRuntime (edu.iu.dsc.tws.executor.core.ExecutionRuntime)8 BlockLocation (edu.iu.dsc.tws.api.data.BlockLocation)7 FSDataOutputStream (edu.iu.dsc.tws.api.data.FSDataOutputStream)7 PrintWriter (java.io.PrintWriter)7 File (java.io.File)6 LocalTextInputPartitioner (edu.iu.dsc.tws.data.api.formatters.LocalTextInputPartitioner)5 Test (org.junit.Test)5 ComputeGraph (edu.iu.dsc.tws.api.compute.graph.ComputeGraph)4 LocalCSVInputPartitioner (edu.iu.dsc.tws.data.api.formatters.LocalCSVInputPartitioner)4 LocalFixedInputPartitioner (edu.iu.dsc.tws.data.api.formatters.LocalFixedInputPartitioner)4 DataGenerator (edu.iu.dsc.tws.tsched.utils.DataGenerator)4 CSVInputSplit (edu.iu.dsc.tws.data.api.splits.CSVInputSplit)3