Examples with HadoopPathBasedBulkFormatBuilder org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder used on opensource projects

Example 1 with HadoopPathBasedBulkFormatBuilder

use of org.apache.flink.streaming.api.functions.sink.filesystem.HadoopPathBasedBulkFormatBuilder in project flink by apache.

the class HadoopPathBasedPartFileWriterTest method testWriteFile.

@Test
public void testWriteFile() throws Exception {
    File file = TEMPORARY_FOLDER.newFolder();
    Path basePath = new Path(file.toURI());
    List<String> data = Arrays.asList("first line", "second line", "third line");
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setParallelism(1);
    env.enableCheckpointing(100);
    DataStream<String> stream = env.addSource(new FiniteTestSource<>(data), TypeInformation.of(String.class));
    Configuration configuration = new Configuration();
    HadoopPathBasedBulkFormatBuilder<String, String, ?> builder = new HadoopPathBasedBulkFormatBuilder<>(basePath, new TestHadoopPathBasedBulkWriterFactory(), configuration, new DateTimeBucketAssigner<>());
    TestStreamingFileSinkFactory<String> streamingFileSinkFactory = new TestStreamingFileSinkFactory<>();
    stream.addSink(streamingFileSinkFactory.createSink(builder, 1000));
    env.execute();
    validateResult(data, configuration, basePath);
}