Examples with CellIndex org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex used on opensource projects

Search in sources :

Example 76 with CellIndex

use of org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex in project incubator-systemml by apache.

the class ABATernaryAggregateTest method runTernaryAggregateTest.

private void runTernaryAggregateTest(String testname, boolean sparse, boolean vectors, boolean rewrites, ExecType et) {
    // rtplatform for MR
    RUNTIME_PLATFORM platformOld = rtplatform;
    switch(et) {
        case MR:
            rtplatform = RUNTIME_PLATFORM.HADOOP;
            break;
        case SPARK:
            rtplatform = RUNTIME_PLATFORM.SPARK;
            break;
        default:
            rtplatform = RUNTIME_PLATFORM.HYBRID;
            break;
    }
    boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG;
    if (rtplatform == RUNTIME_PLATFORM.SPARK)
        DMLScript.USE_LOCAL_SPARK_CONFIG = true;
    boolean rewritesOld = OptimizerUtils.ALLOW_SUM_PRODUCT_REWRITES;
    try {
        TestConfiguration config = getTestConfiguration(testname);
        loadTestConfiguration(config);
        OptimizerUtils.ALLOW_SUM_PRODUCT_REWRITES = rewrites;
        String HOME = SCRIPT_DIR + TEST_DIR;
        fullDMLScriptName = HOME + testname + ".dml";
        programArgs = new String[] { "-explain", "-stats", "-args", input("A"), output("R") };
        fullRScriptName = HOME + testname + ".R";
        rCmd = "Rscript" + " " + fullRScriptName + " " + inputDir() + " " + expectedDir();
        // generate actual dataset
        double sparsity = sparse ? sparsity2 : sparsity1;
        double[][] A = getRandomMatrix(vectors ? rows * cols : rows, vectors ? 1 : cols, 0, 1, sparsity, 17);
        writeInputMatrixWithMTD("A", A, true);
        // run test cases
        runTest(true, false, null, -1);
        runRScript(true);
        // compare output matrices
        HashMap<CellIndex, Double> dmlfile = readDMLMatrixFromHDFS("R");
        HashMap<CellIndex, Double> rfile = readRMatrixFromFS("R");
        TestUtils.compareMatrices(dmlfile, rfile, eps, "Stat-DML", "Stat-R");
        // check for rewritten patterns in statistics output
        if (rewrites && et != ExecType.MR) {
            String opcode = ((et == ExecType.SPARK) ? Instruction.SP_INST_PREFIX : "") + (((testname.equals(TEST_NAME1) || testname.equals(TEST_NAME3) || vectors) ? "tak+*" : "tack+*"));
            Assert.assertTrue(Statistics.getCPHeavyHitterOpCodes().contains(opcode));
        }
    } finally {
        rtplatform = platformOld;
        DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld;
        OptimizerUtils.ALLOW_SUM_PRODUCT_REWRITES = rewritesOld;
    }
}
Also used : RUNTIME_PLATFORM(org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM) CellIndex(org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex) TestConfiguration(org.apache.sysml.test.integration.TestConfiguration)

Example 77 with CellIndex

use of org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex in project incubator-systemml by apache.

the class CovarianceWeightsTest method runCovarianceTest.

/**
 * @param sparseM1
 * @param sparseM2
 * @param instType
 */
private void runCovarianceTest(boolean sparse, ExecType et) {
    // rtplatform for MR
    RUNTIME_PLATFORM platformOld = rtplatform;
    switch(et) {
        case MR:
            rtplatform = RUNTIME_PLATFORM.HADOOP;
            break;
        case SPARK:
            rtplatform = RUNTIME_PLATFORM.SPARK;
            break;
        default:
            rtplatform = RUNTIME_PLATFORM.HYBRID;
            break;
    }
    boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG;
    if (rtplatform == RUNTIME_PLATFORM.SPARK)
        DMLScript.USE_LOCAL_SPARK_CONFIG = true;
    try {
        TestConfiguration config = getTestConfiguration(TEST_NAME);
        loadTestConfiguration(config);
        String HOME = SCRIPT_DIR + TEST_DIR;
        fullDMLScriptName = HOME + TEST_NAME + ".dml";
        programArgs = new String[] { "-args", input("A"), input("B"), input("C"), output("R") };
        fullRScriptName = HOME + TEST_NAME + ".R";
        rCmd = "Rscript" + " " + fullRScriptName + " " + inputDir() + " " + expectedDir();
        // generate actual dataset (always dense because values <=0 invalid)
        double sparsitya = sparse ? sparsity2 : sparsity1;
        double[][] A = getRandomMatrix(rows, 1, 1, maxVal, sparsitya, 7);
        writeInputMatrixWithMTD("A", A, true);
        double[][] B = getRandomMatrix(rows, 1, 1, 1, 1.0, 34);
        writeInputMatrixWithMTD("B", B, true);
        double[][] C = getRandomMatrix(rows, 1, 1, 1, 1.0, 8623);
        writeInputMatrixWithMTD("C", C, true);
        runTest(true, false, null, -1);
        runRScript(true);
        // compare matrices
        HashMap<CellIndex, Double> dmlfile = readDMLMatrixFromHDFS("R");
        HashMap<CellIndex, Double> rfile = readRMatrixFromFS("R");
        TestUtils.compareMatrices(dmlfile, rfile, eps, "Stat-DML", "Stat-R");
    } finally {
        rtplatform = platformOld;
        DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld;
    }
}
Also used : RUNTIME_PLATFORM(org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM) CellIndex(org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex) TestConfiguration(org.apache.sysml.test.integration.TestConfiguration)

Example 78 with CellIndex

use of org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex in project incubator-systemml by apache.

the class FullIfElseTest method runIfElseTest.

private void runIfElseTest(boolean matrix1, boolean matrix2, boolean matrix3, boolean sparse, ExecType et) {
    // rtplatform for MR
    RUNTIME_PLATFORM platformOld = rtplatform;
    switch(et) {
        case MR:
            rtplatform = RUNTIME_PLATFORM.HADOOP;
            break;
        case SPARK:
            rtplatform = RUNTIME_PLATFORM.SPARK;
            break;
        default:
            rtplatform = RUNTIME_PLATFORM.HYBRID_SPARK;
            break;
    }
    boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG;
    boolean rewritesOld = OptimizerUtils.ALLOW_ALGEBRAIC_SIMPLIFICATION;
    if (rtplatform == RUNTIME_PLATFORM.SPARK || rtplatform == RUNTIME_PLATFORM.HYBRID_SPARK)
        DMLScript.USE_LOCAL_SPARK_CONFIG = true;
    // test runtime ops
    OptimizerUtils.ALLOW_ALGEBRAIC_SIMPLIFICATION = false;
    try {
        TestConfiguration config = getTestConfiguration(TEST_NAME1);
        loadTestConfiguration(config);
        String HOME = SCRIPT_DIR + TEST_DIR;
        fullDMLScriptName = HOME + TEST_NAME1 + ".dml";
        programArgs = new String[] { "-explain", "-args", input("A"), input("B"), input("C"), output("R") };
        fullRScriptName = HOME + TEST_NAME1 + ".R";
        rCmd = "Rscript" + " " + fullRScriptName + " " + inputDir() + " " + expectedDir();
        // generate actual datasets (matrices and scalars)
        double sparsity = sparse ? sparsity2 : sparsity1;
        double[][] A = matrix1 ? getRandomMatrix(rows, cols, 0, 1, sparsity, 1) : getScalar(1);
        writeInputMatrixWithMTD("A", A, true);
        double[][] B = matrix2 ? getRandomMatrix(rows, cols, 0, 1, sparsity, 2) : getScalar(2);
        writeInputMatrixWithMTD("B", B, true);
        double[][] C = matrix3 ? getRandomMatrix(rows, cols, 0, 1, sparsity, 3) : getScalar(3);
        writeInputMatrixWithMTD("C", C, true);
        // run test cases
        runTest(true, false, null, -1);
        runRScript(true);
        // compare output matrices
        HashMap<CellIndex, Double> dmlfile = readDMLMatrixFromHDFS("R");
        HashMap<CellIndex, Double> rfile = readRMatrixFromFS("R");
        TestUtils.compareMatrices(dmlfile, rfile, 0, "Stat-DML", "Stat-R");
    } finally {
        rtplatform = platformOld;
        DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld;
        OptimizerUtils.ALLOW_ALGEBRAIC_SIMPLIFICATION = rewritesOld;
    }
}
Also used : RUNTIME_PLATFORM(org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM) CellIndex(org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex) TestConfiguration(org.apache.sysml.test.integration.TestConfiguration)

Example 79 with CellIndex

use of org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex in project incubator-systemml by apache.

the class QuantileWeightsTest method runQuantileTest.

private void runQuantileTest(String TEST_NAME, double p, boolean sparse, ExecType et) {
    // rtplatform for MR
    RUNTIME_PLATFORM platformOld = rtplatform;
    switch(et) {
        case MR:
            rtplatform = RUNTIME_PLATFORM.HADOOP;
            break;
        case SPARK:
            rtplatform = RUNTIME_PLATFORM.SPARK;
            break;
        default:
            rtplatform = RUNTIME_PLATFORM.HYBRID;
            break;
    }
    boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG;
    if (rtplatform == RUNTIME_PLATFORM.SPARK)
        DMLScript.USE_LOCAL_SPARK_CONFIG = true;
    try {
        TestConfiguration config = getTestConfiguration(TEST_NAME);
        loadTestConfiguration(config);
        String HOME = SCRIPT_DIR + TEST_DIR;
        fullDMLScriptName = HOME + TEST_NAME + ".dml";
        programArgs = new String[] { "-args", input("A"), input("W"), Double.toString(p), output("R") };
        fullRScriptName = HOME + TEST_NAME + ".R";
        rCmd = "Rscript" + " " + fullRScriptName + " " + inputDir() + " " + p + " " + expectedDir();
        // generate actual dataset (always dense because values <=0 invalid)
        double sparsitya = sparse ? sparsity2 : sparsity1;
        double[][] A = getRandomMatrix(rows, 1, 1, maxVal, sparsitya, 1236);
        writeInputMatrixWithMTD("A", A, true);
        double[][] W = getRandomMatrix(rows, 1, 1, 1, 1.0, 1);
        writeInputMatrixWithMTD("W", W, true);
        runTest(true, false, null, -1);
        runRScript(true);
        // compare matrices
        HashMap<CellIndex, Double> dmlfile = readDMLMatrixFromHDFS("R");
        HashMap<CellIndex, Double> rfile = readRMatrixFromFS("R");
        TestUtils.compareMatrices(dmlfile, rfile, eps, "Stat-DML", "Stat-R");
    } finally {
        DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld;
        rtplatform = platformOld;
    }
}
Also used : RUNTIME_PLATFORM(org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM) CellIndex(org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex) TestConfiguration(org.apache.sysml.test.integration.TestConfiguration)

Example 80 with CellIndex

use of org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex in project incubator-systemml by apache.

the class TernaryAggregateTest method runTernaryAggregateTest.

private void runTernaryAggregateTest(String testname, boolean sparse, boolean vectors, boolean rewrites, ExecType et) {
    // rtplatform for MR
    RUNTIME_PLATFORM platformOld = rtplatform;
    switch(et) {
        case MR:
            rtplatform = RUNTIME_PLATFORM.HADOOP;
            break;
        case SPARK:
            rtplatform = RUNTIME_PLATFORM.SPARK;
            break;
        default:
            rtplatform = RUNTIME_PLATFORM.HYBRID;
            break;
    }
    boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG;
    if (rtplatform == RUNTIME_PLATFORM.SPARK)
        DMLScript.USE_LOCAL_SPARK_CONFIG = true;
    boolean rewritesOld = OptimizerUtils.ALLOW_SUM_PRODUCT_REWRITES;
    try {
        TestConfiguration config = getTestConfiguration(testname);
        loadTestConfiguration(config);
        OptimizerUtils.ALLOW_SUM_PRODUCT_REWRITES = rewrites;
        String HOME = SCRIPT_DIR + TEST_DIR;
        fullDMLScriptName = HOME + testname + ".dml";
        programArgs = new String[] { "-stats", "-args", input("A"), output("R") };
        fullRScriptName = HOME + testname + ".R";
        rCmd = "Rscript" + " " + fullRScriptName + " " + inputDir() + " " + expectedDir();
        // generate actual dataset
        double sparsity = sparse ? sparsity2 : sparsity1;
        double[][] A = getRandomMatrix(vectors ? rows * cols : rows, vectors ? 1 : cols, 0, 1, sparsity, 17);
        writeInputMatrixWithMTD("A", A, true);
        // run test cases
        runTest(true, false, null, -1);
        runRScript(true);
        // compare output matrices
        HashMap<CellIndex, Double> dmlfile = readDMLMatrixFromHDFS("R");
        HashMap<CellIndex, Double> rfile = readRMatrixFromFS("R");
        TestUtils.compareMatrices(dmlfile, rfile, eps, "Stat-DML", "Stat-R");
        // check for rewritten patterns in statistics output
        if (rewrites && et != ExecType.MR) {
            String opcode = ((et == ExecType.SPARK) ? Instruction.SP_INST_PREFIX : "") + (((testname.equals(TEST_NAME1) || vectors) ? "tak+*" : "tack+*"));
            Assert.assertEquals(Boolean.TRUE, Boolean.valueOf(Statistics.getCPHeavyHitterOpCodes().contains(opcode)));
        }
    } finally {
        rtplatform = platformOld;
        DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld;
        OptimizerUtils.ALLOW_SUM_PRODUCT_REWRITES = rewritesOld;
    }
}
Also used : RUNTIME_PLATFORM(org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM) CellIndex(org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex) TestConfiguration(org.apache.sysml.test.integration.TestConfiguration)

Aggregations

CellIndex (org.apache.sysml.runtime.matrix.data.MatrixValue.CellIndex)257 TestConfiguration (org.apache.sysml.test.integration.TestConfiguration)201 RUNTIME_PLATFORM (org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM)173 MatrixCharacteristics (org.apache.sysml.runtime.matrix.MatrixCharacteristics)37 Test (org.junit.Test)20 HashMap (java.util.HashMap)16 IOException (java.io.IOException)14 MatrixBlock (org.apache.sysml.runtime.matrix.data.MatrixBlock)9 Random (java.util.Random)8 ArrayList (java.util.ArrayList)7 BufferedReader (java.io.BufferedReader)6 StringTokenizer (java.util.StringTokenizer)6 InputStreamReader (java.io.InputStreamReader)5 FSDataInputStream (org.apache.hadoop.fs.FSDataInputStream)5 FileSystem (org.apache.hadoop.fs.FileSystem)5 Path (org.apache.hadoop.fs.Path)5 FileStatus (org.apache.hadoop.fs.FileStatus)4 FileReader (java.io.FileReader)2 DMLException (org.apache.sysml.api.DMLException)2 MMultMethod (org.apache.sysml.hops.AggBinaryOp.MMultMethod)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial