Search in sources :

Example 1 with CassandraPojoOutputFormat

use of org.apache.flink.batch.connectors.cassandra.CassandraPojoOutputFormat in project flink by apache.

the class BatchPojoExample method main.

public static void main(String[] args) throws Exception {
    ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(1);
    List<Pojo> customCassandraAnnotatedPojos = IntStream.range(0, 20).mapToObj(x -> new Pojo(UUID.randomUUID().toString(), x, 0)).collect(Collectors.toList());
    DataSet<Pojo> dataSet = env.fromCollection(customCassandraAnnotatedPojos);
    ClusterBuilder clusterBuilder = new ClusterBuilder() {

        private static final long serialVersionUID = -1754532803757154795L;

        @Override
        protected Cluster buildCluster(Cluster.Builder builder) {
            return builder.addContactPoints("127.0.0.1").build();
        }
    };
    dataSet.output(new CassandraPojoOutputFormat<>(clusterBuilder, Pojo.class, () -> new Mapper.Option[] { Mapper.Option.saveNullFields(true) }));
    env.execute("Write");
    /*
         *	This is for the purpose of showing an example of creating a DataSet using CassandraPojoInputFormat.
         */
    DataSet<Pojo> inputDS = env.createInput(new CassandraPojoInputFormat<>(SELECT_QUERY, clusterBuilder, Pojo.class, () -> new Mapper.Option[] { Mapper.Option.consistencyLevel(ConsistencyLevel.ANY) }));
    inputDS.print();
}
Also used : ClusterBuilder(org.apache.flink.streaming.connectors.cassandra.ClusterBuilder) IntStream(java.util.stream.IntStream) DataSet(org.apache.flink.api.java.DataSet) List(java.util.List) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) CassandraPojoOutputFormat(org.apache.flink.batch.connectors.cassandra.CassandraPojoOutputFormat) Cluster(com.datastax.driver.core.Cluster) Mapper(com.datastax.driver.mapping.Mapper) UUID(java.util.UUID) Collectors(java.util.stream.Collectors) CassandraPojoInputFormat(org.apache.flink.batch.connectors.cassandra.CassandraPojoInputFormat) ConsistencyLevel(com.datastax.driver.core.ConsistencyLevel) ExecutionEnvironment(org.apache.flink.api.java.ExecutionEnvironment) ClusterBuilder(org.apache.flink.streaming.connectors.cassandra.ClusterBuilder) ClusterBuilder(org.apache.flink.streaming.connectors.cassandra.ClusterBuilder)

Example 2 with CassandraPojoOutputFormat

use of org.apache.flink.batch.connectors.cassandra.CassandraPojoOutputFormat in project flink by apache.

the class CassandraConnectorITCase method writePojosWithOutputFormat.

private <T> List<T> writePojosWithOutputFormat(Class<T> annotatedPojoClass) throws Exception {
    final CassandraPojoOutputFormat<T> sink = new CassandraPojoOutputFormat<>(builderForWriting, annotatedPojoClass, () -> new Mapper.Option[] { Mapper.Option.saveNullFields(true) });
    final Constructor<T> pojoConstructor = getPojoConstructor(annotatedPojoClass);
    List<T> pojos = new ArrayList<>();
    for (int i = 0; i < 20; i++) {
        pojos.add(pojoConstructor.newInstance(UUID.randomUUID().toString(), i, 0));
    }
    try {
        sink.configure(new Configuration());
        sink.open(0, 1);
        for (T pojo : pojos) {
            sink.writeRecord(pojo);
        }
    } finally {
        sink.close();
    }
    return pojos;
}
Also used : CassandraPojoOutputFormat(org.apache.flink.batch.connectors.cassandra.CassandraPojoOutputFormat) Mapper(com.datastax.driver.mapping.Mapper) Configuration(org.apache.flink.configuration.Configuration) ArrayList(java.util.ArrayList)

Aggregations

Mapper (com.datastax.driver.mapping.Mapper)2 CassandraPojoOutputFormat (org.apache.flink.batch.connectors.cassandra.CassandraPojoOutputFormat)2 Cluster (com.datastax.driver.core.Cluster)1 ConsistencyLevel (com.datastax.driver.core.ConsistencyLevel)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1 UUID (java.util.UUID)1 Collectors (java.util.stream.Collectors)1 IntStream (java.util.stream.IntStream)1 DataSet (org.apache.flink.api.java.DataSet)1 ExecutionEnvironment (org.apache.flink.api.java.ExecutionEnvironment)1 CassandraPojoInputFormat (org.apache.flink.batch.connectors.cassandra.CassandraPojoInputFormat)1 Configuration (org.apache.flink.configuration.Configuration)1 ClusterBuilder (org.apache.flink.streaming.connectors.cassandra.ClusterBuilder)1