Search in sources :

Example 1 with ArrowFileReader

use of org.apache.arrow.vector.ipc.ArrowFileReader in project twister2 by DSC-SPIDAL.

the class Twister2ArrowFileReader method initInputFile.

public void initInputFile() {
    try {
        LOG.info("arrow schema:" + Schema.fromJSON(arrowSchema));
        Path path = new Path(arrowInputFile);
        this.fileSystem = FileSystemUtils.get(path);
        this.fsDataInputStream = fileSystem.open(path);
        this.fileInputStream = new FileInputStream(arrowInputFile);
        this.arrowFileReader = new ArrowFileReader(new SeekableReadChannel(fileInputStream.getChannel()), rootAllocator);
        this.root = arrowFileReader.getVectorSchemaRoot();
        arrowBlocks = arrowFileReader.getRecordBlocks();
        LOG.info("\nReading the arrow file : " + arrowInputFile + "\tFile size:" + arrowInputFile.length() + "\tschema:" + root.getSchema().toString() + "\tArrow Blocks Size: " + arrowBlocks.size());
    } catch (FileNotFoundException e) {
        throw new Twister2RuntimeException("File Not Found", e);
    } catch (Exception ioe) {
        throw new Twister2RuntimeException("IOException Occured", ioe);
    }
}
Also used : Path(edu.iu.dsc.tws.api.data.Path) Twister2RuntimeException(edu.iu.dsc.tws.api.exceptions.Twister2RuntimeException) ArrowFileReader(org.apache.arrow.vector.ipc.ArrowFileReader) FileNotFoundException(java.io.FileNotFoundException) SeekableReadChannel(org.apache.arrow.vector.ipc.SeekableReadChannel) FileInputStream(java.io.FileInputStream) IOException(java.io.IOException) FileNotFoundException(java.io.FileNotFoundException) Twister2RuntimeException(edu.iu.dsc.tws.api.exceptions.Twister2RuntimeException)

Aggregations

Path (edu.iu.dsc.tws.api.data.Path)1 Twister2RuntimeException (edu.iu.dsc.tws.api.exceptions.Twister2RuntimeException)1 FileInputStream (java.io.FileInputStream)1 FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 ArrowFileReader (org.apache.arrow.vector.ipc.ArrowFileReader)1 SeekableReadChannel (org.apache.arrow.vector.ipc.SeekableReadChannel)1