Examples with JdbcRDD org.apache.spark.rdd.JdbcRDD used on opensource projects

Example 1 with JdbcRDD

use of org.apache.spark.rdd.JdbcRDD in project kylo by Teradata.

the class SqlTransformStage method extractData.

@Override
public TransformResult extractData(@Nonnull final ResultSet rs) throws SQLException {
    final ResultSetMetaData metaData = rs.getMetaData();
    final TransformResult result = new TransformResult();
    final StructType schema = extractSchema(metaData, result);
    // Create data set
    final Function0<Connection> getConnection = ScalaUtil.wrap(Suppliers.compose(JdbcUtil.getDataSourceConnection(), dataSource));
    final Function1<ResultSet, Row> mapRow = ScalaUtil.wrap(new RowTransform());
    // noinspection RedundantCast,unchecked
    final ClassTag<Row> classTag = (ClassTag) ClassTag$.MODULE$.apply(Row.class);
    final RDD<Row> rdd = new JdbcRDD<Row>(sqlContext.sparkContext(), getConnection, "SELECT * FROM (" + sql + ") rdd WHERE ? = ?", 1, 1, 1, mapRow, classTag);
    result.setDataSet(sparkContextService.toDataSet(sqlContext, rdd.toJavaRDD(), schema));
    return result;
}