Examples with SparkConf org.apache.spark.SparkConf used on opensource projects

Example 6 with SparkConf

use of org.apache.spark.SparkConf in project deeplearning4j by deeplearning4j.

the class BaseSparkTest method getContext.

/**
     *
     * @return
     */
public JavaSparkContext getContext() {
    if (sc != null)
        return sc;
    // set to test mode
    SparkConf sparkConf = new SparkConf().setMaster("local[" + numExecutors() + "]").setAppName("sparktest");
    sc = new JavaSparkContext(sparkConf);
    return sc;
}

Example 7 with SparkConf

use of org.apache.spark.SparkConf in project deeplearning4j by deeplearning4j.

the class TestKryoWarning method testKryoMessageMLNCorrectConfigKryo.

@Test
@Ignore
public void testKryoMessageMLNCorrectConfigKryo() {
    //Should NOT print warning message
    SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("sparktest").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").set("spark.kryo.registrator", "org.nd4j.Nd4jRegistrator");
    doTestMLN(sparkConf);
}

Example 8 with SparkConf

use of org.apache.spark.SparkConf in project deeplearning4j by deeplearning4j.

the class TestKryoWarning method testKryoMessageMLNCorrectConfigNoKryo.

@Test
@Ignore
public void testKryoMessageMLNCorrectConfigNoKryo() {
    //Should NOT print warning message
    SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("sparktest");
    doTestMLN(sparkConf);
}

Example 9 with SparkConf

use of org.apache.spark.SparkConf in project deeplearning4j by deeplearning4j.

the class TestKryoWarning method testKryoMessageCGIncorrectConfig.

@Test
@Ignore
public void testKryoMessageCGIncorrectConfig() {
    //Should print warning message
    SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("sparktest").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
    doTestCG(sparkConf);
}

Example 10 with SparkConf

use of org.apache.spark.SparkConf in project deeplearning4j by deeplearning4j.

the class TestKryoWarning method testKryoMessageCGCorrectConfigKryo.

@Test
@Ignore
public void testKryoMessageCGCorrectConfigKryo() {
    //Should NOT print warning message
    SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("sparktest").set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").set("spark.kryo.registrator", "org.nd4j.Nd4jRegistrator");
    doTestCG(sparkConf);
}