Example 56 with StructType
use of org.apache.spark.sql.types.StructType in project net.jgp.labs.spark by jgperrin.
the class FirstPrediction method start.
private void start() {
SparkSession spark = SparkSession.builder().appName("First Prediction").master("local").getOrCreate();
StructType schema = new StructType(new StructField[] { new StructField("label", DataTypes.DoubleType, false, Metadata.empty()), new StructField("features", new VectorUDT(), false, Metadata.empty()) });
// TODO this example is not working yet
}
Also used :
VectorUDT(org.apache.spark.mllib.linalg.VectorUDT)
SparkSession(org.apache.spark.sql.SparkSession)
StructField(org.apache.spark.sql.types.StructField)
StructType(org.apache.spark.sql.types.StructType)
Aggregations
StructType (org.apache.spark.sql.types.StructType)56
Row (org.apache.spark.sql.Row)49
StructField (org.apache.spark.sql.types.StructField)48
ArrayList (java.util.ArrayList)43
Test (org.junit.Test)37
Script (org.apache.sysml.api.mlcontext.Script)35
VectorUDT (org.apache.spark.ml.linalg.VectorUDT)17
MatrixMetadata (org.apache.sysml.api.mlcontext.MatrixMetadata)17
DenseVector (org.apache.spark.ml.linalg.DenseVector)15
Vector (org.apache.spark.ml.linalg.Vector)14
SparkSession (org.apache.spark.sql.SparkSession)11
MLResults (org.apache.sysml.api.mlcontext.MLResults)6
MatrixBlock (org.apache.sysml.runtime.matrix.data.MatrixBlock)6
Tuple2 (scala.Tuple2)6
JavaSparkContext (org.apache.spark.api.java.JavaSparkContext)5
FrameMetadata (org.apache.sysml.api.mlcontext.FrameMetadata)5
DataType (org.apache.spark.sql.types.DataType)4
JavaRDD (org.apache.spark.api.java.JavaRDD)3
ValueType (org.apache.sysml.parser.Expression.ValueType)3
CommaSeparatedValueStringToDoubleArrayRow (org.apache.sysml.test.integration.mlcontext.MLContextTest.CommaSeparatedValueStringToDoubleArrayRow)3
