Search in sources :

Example 26 with FileSystem

use of edu.iu.dsc.tws.api.data.FileSystem in project twister2 by DSC-SPIDAL.

the class DataGenerator method generate.

public void generate(Path directory, int size, int dimension) {
    try {
        FileSystem fs = FileSystemUtils.get(directory.toUri(), config);
        if (fs.exists(directory)) {
            fs.delete(directory, true);
        }
        FSDataOutputStream outputStream = fs.create(new Path(directory, generateRandom(10) + ".txt"));
        PrintWriter pw = new PrintWriter(outputStream);
        String points = generatePoints(size, dimension, 100);
        pw.print(points);
        outputStream.sync();
        pw.close();
    } catch (IOException e) {
        throw new RuntimeException("Data Generation Error Occured", e);
    }
}
Also used : Path(edu.iu.dsc.tws.api.data.Path) FileSystem(edu.iu.dsc.tws.api.data.FileSystem) FSDataOutputStream(edu.iu.dsc.tws.api.data.FSDataOutputStream) IOException(java.io.IOException) PrintWriter(java.io.PrintWriter)

Aggregations

FileSystem (edu.iu.dsc.tws.api.data.FileSystem)26 FileStatus (edu.iu.dsc.tws.api.data.FileStatus)19 Path (edu.iu.dsc.tws.api.data.Path)18 ArrayList (java.util.ArrayList)11 IOException (java.io.IOException)10 BlockLocation (edu.iu.dsc.tws.api.data.BlockLocation)7 FileInputSplit (edu.iu.dsc.tws.data.api.splits.FileInputSplit)7 FSDataOutputStream (edu.iu.dsc.tws.api.data.FSDataOutputStream)4 PrintWriter (java.io.PrintWriter)4 CSVInputSplit (edu.iu.dsc.tws.data.api.splits.CSVInputSplit)3 LocalFileSystem (edu.iu.dsc.tws.data.fs.local.LocalFileSystem)3 HadoopFileSystem (edu.iu.dsc.tws.data.hdfs.HadoopFileSystem)3 File (java.io.File)2 URI (java.net.URI)2 URISyntaxException (java.net.URISyntaxException)2 Random (java.util.Random)2 TaskSchedulerException (edu.iu.dsc.tws.api.compute.exceptions.TaskSchedulerException)1 Config (edu.iu.dsc.tws.api.config.Config)1 BinaryInputSplit (edu.iu.dsc.tws.data.api.splits.BinaryInputSplit)1 DataFileReader (edu.iu.dsc.tws.data.utils.DataFileReader)1