Search in sources :

Example 46 with DataFileReader

use of org.apache.avro.file.DataFileReader in project mist by snuspl.

the class AvroExecutionVertexStore method loadAvroPhysicalSourceOutgoingEdgesInfoFromFile.

// TODO: [MIST-*] Implement policy for deleting from disk. Currently does not delete from disk.
private AvroPhysicalSourceOutgoingEdgesInfo loadAvroPhysicalSourceOutgoingEdgesInfoFromFile(final File storedSource) {
    try {
        final DataFileReader<AvroPhysicalSourceOutgoingEdgesInfo> dataFileReader = new DataFileReader<>(storedSource, sourceDatumReader);
        AvroPhysicalSourceOutgoingEdgesInfo avroPhysicalSourceOutgoingEdgesInfo = null;
        avroPhysicalSourceOutgoingEdgesInfo = dataFileReader.next(avroPhysicalSourceOutgoingEdgesInfo);
        return avroPhysicalSourceOutgoingEdgesInfo;
    } catch (IOException e) {
        throw new RuntimeException("Loading AvroPhysicalSourceOutgoingEdgesInfo has failed.");
    }
}
Also used : DataFileReader(org.apache.avro.file.DataFileReader) IOException(java.io.IOException) AvroPhysicalSourceOutgoingEdgesInfo(edu.snu.mist.formats.avro.AvroPhysicalSourceOutgoingEdgesInfo)

Aggregations

DataFileReader (org.apache.avro.file.DataFileReader)46 GenericRecord (org.apache.avro.generic.GenericRecord)28 File (java.io.File)26 GenericDatumReader (org.apache.avro.generic.GenericDatumReader)21 Schema (org.apache.avro.Schema)20 Test (org.junit.Test)10 ArrayList (java.util.ArrayList)9 IOException (java.io.IOException)8 Test (org.testng.annotations.Test)7 SeekableInput (org.apache.avro.file.SeekableInput)6 WorkUnitState (org.apache.gobblin.configuration.WorkUnitState)6 Configuration (org.apache.hadoop.conf.Configuration)6 ReflectDatumReader (org.apache.avro.reflect.ReflectDatumReader)5 SeekableByteArrayInput (org.apache.avro.file.SeekableByteArrayInput)4 FsInput (org.apache.avro.mapred.FsInput)4 SpecificDatumReader (org.apache.avro.specific.SpecificDatumReader)4 Utf8 (org.apache.avro.util.Utf8)4 JsonObject (com.google.gson.JsonObject)2 AvroDag (edu.snu.mist.formats.avro.AvroDag)2 Date (java.sql.Date)2