Examples with MatrixReader org.apache.sysml.runtime.io.MatrixReader used on opensource projects

Example 6 with MatrixReader

use of org.apache.sysml.runtime.io.MatrixReader in project incubator-systemml by apache.

the class WriteTest method testBinary.

@Test
public void testBinary() {
    int rows = 10;
    int cols = 10;
    TestConfiguration config = availableTestConfigurations.get("BinaryTest");
    config.addVariable("rows", rows);
    config.addVariable("cols", cols);
    config.addVariable("format", "binary");
    loadTestConfiguration(config);
    double[][] a = getRandomMatrix(rows, cols, -1, 1, 0.7, System.currentTimeMillis());
    writeInputMatrixWithMTD("a", a, false, new MatrixCharacteristics(rows, cols, 1000, 1000));
    runTest();
    //read and compare output matrix
    try {
        MatrixReader reader = MatrixReaderFactory.createMatrixReader(InputInfo.BinaryBlockInputInfo);
        MatrixBlock mb = reader.readMatrixFromHDFS(output("a"), rows, cols, 1000, 1000, -1);
        checkDMLMetaDataFile("a", new MatrixCharacteristics(rows, cols, 1000, 1000));
        TestUtils.compareMatrices(a, DataConverter.convertToDoubleMatrix(mb), rows, cols, 0);
    } catch (Exception ex) {
        throw new RuntimeException(ex);
    }
}

Example 7 with MatrixReader

use of org.apache.sysml.runtime.io.MatrixReader in project incubator-systemml by apache.

the class DataConverter method readMatrixFromHDFS.

/**
	 * Core method for reading matrices in format textcell, matrixmarket, binarycell, or binaryblock 
	 * from HDFS into main memory. For expected dense matrices we directly copy value- or block-at-a-time 
	 * into the target matrix. In contrast, for sparse matrices, we append (column-value)-pairs and do a 
	 * final sort if required in order to prevent large reorg overheads and increased memory consumption 
	 * in case of unordered inputs.  
	 * 
	 * DENSE MxN input:
	 *  * best/average/worst: O(M*N)
	 * SPARSE MxN input
	 *  * best (ordered, or binary block w/ clen<=bclen): O(M*N)
	 *  * average (unordered): O(M*N*log(N))
	 *  * worst (descending order per row): O(M * N^2)
	 * 
	 * NOTE: providing an exact estimate of 'expected sparsity' can prevent a full copy of the result
	 * matrix block (required for changing sparse->dense, or vice versa)
	 * 
	 * @param prop read properties
	 * @return matrix block
	 * @throws IOException if IOException occurs
	 */
public static MatrixBlock readMatrixFromHDFS(ReadProperties prop) throws IOException {
    //Timing time = new Timing(true);
    long estnnz = (long) (prop.expectedSparsity * prop.rlen * prop.clen);
    //core matrix reading 
    MatrixBlock ret = null;
    try {
        MatrixReader reader = MatrixReaderFactory.createMatrixReader(prop);
        ret = reader.readMatrixFromHDFS(prop.path, prop.rlen, prop.clen, prop.brlen, prop.bclen, estnnz);
    } catch (DMLRuntimeException rex) {
        throw new IOException(rex);
    }
    return ret;
}

Example 8 with MatrixReader

use of org.apache.sysml.runtime.io.MatrixReader in project incubator-systemml by apache.

the class MapReduceTool method readColumnVectorFromHDFS.

public static double[] readColumnVectorFromHDFS(String dir, InputInfo inputinfo, long rlen, long clen, int brlen, int bclen) throws IOException, DMLRuntimeException {
    MatrixReader reader = MatrixReaderFactory.createMatrixReader(inputinfo);
    MatrixBlock mb = reader.readMatrixFromHDFS(dir, rlen, clen, brlen, bclen, rlen * clen);
    return DataConverter.convertToDoubleVector(mb);
}

Example 9 with MatrixReader

use of org.apache.sysml.runtime.io.MatrixReader in project incubator-systemml by apache.

the class Connection method readDoubleMatrix.

/**
	 * Reads an input matrix in arbitrary format from HDFS into a dense double array.
	 * NOTE: this call currently only supports default configurations for CSV.
	 * 
	 * @param fname the filename of the input matrix
	 * @param iinfo InputInfo object
	 * @param rows number of rows in the matrix
	 * @param cols number of columns in the matrix
	 * @param brlen number of rows per block
	 * @param bclen number of columns per block
	 * @param nnz number of non-zero values, -1 indicates unknown
	 * @return matrix as a two-dimensional double array
	 * @throws IOException if IOException occurs
	 */
public double[][] readDoubleMatrix(String fname, InputInfo iinfo, long rows, long cols, int brlen, int bclen, long nnz) throws IOException {
    try {
        MatrixReader reader = MatrixReaderFactory.createMatrixReader(iinfo);
        MatrixBlock mb = reader.readMatrixFromHDFS(fname, rows, cols, brlen, bclen, nnz);
        return DataConverter.convertToDoubleMatrix(mb);
    } catch (Exception ex) {
        throw new IOException(ex);
    }
}

Example 10 with MatrixReader

use of org.apache.sysml.runtime.io.MatrixReader in project incubator-systemml by apache.

the class FrameMatrixReblockTest method readMatrixOutput.

/**
	 * 
	 * @param fname
	 * @param rows
	 * @param cols
	 * @param ofmt
	 * @return
	 * @throws DMLRuntimeException
	 * @throws IOException
	 */
private double[][] readMatrixOutput(String fname, String ofmt, int rows, int cols) throws DMLRuntimeException, IOException {
    MatrixReader reader = MatrixReaderFactory.createMatrixReader(InputInfo.stringExternalToInputInfo(ofmt));
    MatrixBlock mb = reader.readMatrixFromHDFS(fname, rows, cols, 1000, 1000, -1);
    return DataConverter.convertToDoubleMatrix(mb);
}