Examples with FrameBlock org.apache.sysml.runtime.matrix.data.FrameBlock used on opensource projects

Example 76 with FrameBlock

use of org.apache.sysml.runtime.matrix.data.FrameBlock in project incubator-systemml by apache.

the class ConvertFrameBlockToIJVLines method call.

@Override
public Iterator<String> call(Tuple2<Long, FrameBlock> kv) throws Exception {
    long rowoffset = kv._1;
    FrameBlock block = kv._2;
    ArrayList<String> cells = new ArrayList<>();
    // write frame meta data
    if (rowoffset == 1) {
        for (int j = 0; j < block.getNumColumns(); j++) if (!block.isColumnMetadataDefault(j)) {
            cells.add("-1 " + (j + 1) + " " + block.getColumnMetadata(j).getNumDistinct());
            cells.add("-2 " + (j + 1) + " " + block.getColumnMetadata(j).getMvValue());
        }
    }
    // convert frame block to list of ijv cell triples
    StringBuilder sb = new StringBuilder();
    Iterator<String[]> iter = block.getStringRowIterator();
    for (int i = 0; iter.hasNext(); i++) {
        // for all rows
        String rowIndex = Long.toString(rowoffset + i);
        String[] row = iter.next();
        for (int j = 0; j < row.length; j++) {
            if (row[j] != null) {
                sb.append(rowIndex);
                sb.append(' ');
                sb.append(j + 1);
                sb.append(' ');
                sb.append(row[j]);
                cells.add(sb.toString());
                sb.setLength(0);
            }
        }
    }
    return cells.iterator();
}

Example 77 with FrameBlock

use of org.apache.sysml.runtime.matrix.data.FrameBlock in project incubator-systemml by apache.

the class FrameReader method createOutputFrameBlock.

/**
 * NOTE: mallocDense controls if the output matrix blocks is fully allocated, this can be redundant
 * if binary block read and single block.
 *
 * @param schema schema as array of ValueTypes
 * @param names column names
 * @param nrow number of rows
 * @return frame block
 * @throws IOException if IOException occurs
 */
protected static FrameBlock createOutputFrameBlock(ValueType[] schema, String[] names, long nrow) throws IOException {
    // check schema and column names
    if (!OptimizerUtils.isValidCPDimensions(schema, names))
        throw new DMLRuntimeException("Schema and names to be define with equal size.");
    // prepare result frame block
    FrameBlock ret = new FrameBlock(schema, names);
    ret.ensureAllocatedColumns((int) nrow);
    return ret;
}

Example 78 with FrameBlock

use of org.apache.sysml.runtime.matrix.data.FrameBlock in project incubator-systemml by apache.

the class FrameReaderBinaryBlock method readFrameFromHDFS.

@Override
public final FrameBlock readFrameFromHDFS(String fname, ValueType[] schema, String[] names, long rlen, long clen) throws IOException, DMLRuntimeException {
    // allocate output frame block
    ValueType[] lschema = createOutputSchema(schema, clen);
    String[] lnames = createOutputNames(names, clen);
    FrameBlock ret = createOutputFrameBlock(lschema, lnames, rlen);
    // prepare file access
    JobConf job = new JobConf(ConfigurationManager.getCachedJobConf());
    Path path = new Path(fname);
    FileSystem fs = IOUtilFunctions.getFileSystem(path, job);
    // check existence and non-empty file
    checkValidInputFile(fs, path);
    // core read (sequential/parallel)
    readBinaryBlockFrameFromHDFS(path, job, fs, ret, rlen, clen);
    return ret;
}

Example 79 with FrameBlock

use of org.apache.sysml.runtime.matrix.data.FrameBlock in project incubator-systemml by apache.

the class FrameReaderBinaryBlock method readFirstBlock.

/**
 * Specific functionality of FrameReaderBinaryBlock, mostly used for testing.
 *
 * @param fname file name
 * @return frame block
 * @throws IOException if IOException occurs
 */
@SuppressWarnings("deprecation")
public FrameBlock readFirstBlock(String fname) throws IOException {
    // prepare file access
    JobConf job = new JobConf(ConfigurationManager.getCachedJobConf());
    Path path = new Path(fname);
    FileSystem fs = IOUtilFunctions.getFileSystem(path, job);
    LongWritable key = new LongWritable();
    FrameBlock value = new FrameBlock();
    // read first block from first file
    Path lpath = IOUtilFunctions.getSequenceFilePaths(fs, path)[0];
    SequenceFile.Reader reader = new SequenceFile.Reader(fs, lpath, job);
    try {
        reader.next(key, value);
    } finally {
        IOUtilFunctions.closeSilently(reader);
    }
    return value;
}

Example 80 with FrameBlock

use of org.apache.sysml.runtime.matrix.data.FrameBlock in project incubator-systemml by apache.

the class FrameWriter method createFrameBlocksForReuse.

public static FrameBlock[] createFrameBlocksForReuse(ValueType[] schema, String[] names, long rlen) {
    FrameBlock[] frameBlock = new FrameBlock[1];
    frameBlock[0] = new FrameBlock(schema, names);
    return frameBlock;
}