Search in sources :

Example 1 with JSONException

use of org.apache.wink.json4j.JSONException in project incubator-systemml by apache.

the class ApplyTfBBMapper method configure.

@Override
public void configure(JobConf job) {
    super.configure(job);
    try {
        _partFileWithHeader = TfUtils.isPartFileWithHeader(job);
        tfmapper = new TfUtils(job);
        tfmapper.loadTfMetadata(job, true);
        // Load relevant information for CSV Reblock
        ByteWritable key = new ByteWritable();
        OffsetCount value = new OffsetCount();
        Path p = new Path(job.get(CSVReblockMR.ROWID_FILE_NAME));
        Path path = new Path(job.get(MRConfigurationNames.MR_MAP_INPUT_FILE));
        FileSystem fs = IOUtilFunctions.getFileSystem(path, job);
        String thisfile = path.makeQualified(fs).toString();
        SequenceFile.Reader reader = null;
        try {
            reader = new SequenceFile.Reader(fs, p, job);
            while (reader.next(key, value)) {
                // "key" needn't be checked since the offset file has information about a single CSV input (the raw data file)
                if (thisfile.equals(value.filename))
                    offsetMap.put(value.fileOffset, value.count);
            }
        } finally {
            IOUtilFunctions.closeSilently(reader);
        }
        idxRow = new CSVReblockMapper.IndexedBlockRow();
        int maxBclen = 0;
        for (ArrayList<CSVReblockInstruction> insv : csv_reblock_instructions) for (CSVReblockInstruction in : insv) {
            if (maxBclen < in.bclen)
                maxBclen = in.bclen;
        }
        //always dense since common csv usecase
        idxRow.getRow().data.reset(1, maxBclen, false);
    } catch (IOException e) {
        throw new RuntimeException(e);
    } catch (JSONException e) {
        throw new RuntimeException(e);
    }
}
Also used : Path(org.apache.hadoop.fs.Path) CSVReblockMapper(org.apache.sysml.runtime.matrix.mapred.CSVReblockMapper) IndexedBlockRow(org.apache.sysml.runtime.matrix.mapred.CSVReblockMapper.IndexedBlockRow) CSVReblockInstruction(org.apache.sysml.runtime.instructions.mr.CSVReblockInstruction) JSONException(org.apache.wink.json4j.JSONException) IOException(java.io.IOException) OffsetCount(org.apache.sysml.runtime.matrix.CSVReblockMR.OffsetCount) DMLRuntimeException(org.apache.sysml.runtime.DMLRuntimeException) SequenceFile(org.apache.hadoop.io.SequenceFile) FileSystem(org.apache.hadoop.fs.FileSystem) ByteWritable(org.apache.hadoop.io.ByteWritable)

Example 2 with JSONException

use of org.apache.wink.json4j.JSONException in project incubator-systemml by apache.

the class ApplyTfCSVMapper method configure.

@Override
public void configure(JobConf job) {
    try {
        _rJob = job;
        _partFileWithHeader = TfUtils.isPartFileWithHeader(job);
        tfmapper = new TfUtils(job);
        tfmapper.loadTfMetadata(job, true);
    } catch (IOException e) {
        throw new RuntimeException(e);
    } catch (JSONException e) {
        throw new RuntimeException(e);
    }
}
Also used : DMLRuntimeException(org.apache.sysml.runtime.DMLRuntimeException) JSONException(org.apache.wink.json4j.JSONException) IOException(java.io.IOException)

Aggregations

IOException (java.io.IOException)2 DMLRuntimeException (org.apache.sysml.runtime.DMLRuntimeException)2 JSONException (org.apache.wink.json4j.JSONException)2 FileSystem (org.apache.hadoop.fs.FileSystem)1 Path (org.apache.hadoop.fs.Path)1 ByteWritable (org.apache.hadoop.io.ByteWritable)1 SequenceFile (org.apache.hadoop.io.SequenceFile)1 CSVReblockInstruction (org.apache.sysml.runtime.instructions.mr.CSVReblockInstruction)1 OffsetCount (org.apache.sysml.runtime.matrix.CSVReblockMR.OffsetCount)1 CSVReblockMapper (org.apache.sysml.runtime.matrix.mapred.CSVReblockMapper)1 IndexedBlockRow (org.apache.sysml.runtime.matrix.mapred.CSVReblockMapper.IndexedBlockRow)1