Search in sources :

Example 11 with OutputFileConfig

use of org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig in project flink by apache.

the class FileWriterTest method testPreCommit.

@Test
public void testPreCommit() throws Exception {
    File outDir = TEMP_FOLDER.newFolder();
    Path path = new Path(outDir.toURI());
    FileWriter<String> fileWriter = createWriter(path, OnCheckpointRollingPolicy.build(), new OutputFileConfig("part-", ""));
    fileWriter.write("test1", new ContextImpl());
    fileWriter.write("test1", new ContextImpl());
    fileWriter.write("test2", new ContextImpl());
    fileWriter.write("test2", new ContextImpl());
    fileWriter.write("test3", new ContextImpl());
    Collection<FileSinkCommittable> committables = fileWriter.prepareCommit();
    assertEquals(3, committables.size());
}
Also used : Path(org.apache.flink.core.fs.Path) OutputFileConfig(org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig) FileSinkCommittable(org.apache.flink.connector.file.sink.FileSinkCommittable) File(java.io.File) Test(org.junit.Test)

Aggregations

OutputFileConfig (org.apache.flink.streaming.api.functions.sink.filesystem.OutputFileConfig)11 File (java.io.File)7 Path (org.apache.flink.core.fs.Path)7 Test (org.junit.Test)6 PartitionCommitInfo (org.apache.flink.connector.file.table.stream.PartitionCommitInfo)3 HiveWriterFactory (org.apache.flink.connectors.hive.write.HiveWriterFactory)3 RowData (org.apache.flink.table.data.RowData)3 IOException (java.io.IOException)2 UncheckedIOException (java.io.UncheckedIOException)2 Properties (java.util.Properties)2 FileSystemTableSink (org.apache.flink.connector.file.table.FileSystemTableSink)2 TableBucketAssigner (org.apache.flink.connector.file.table.FileSystemTableSink.TableBucketAssigner)2 CompactReader (org.apache.flink.connector.file.table.stream.compact.CompactReader)2 HiveCompactReaderFactory (org.apache.flink.connectors.hive.read.HiveCompactReaderFactory)2 HiveBulkWriterFactory (org.apache.flink.connectors.hive.write.HiveBulkWriterFactory)2 HiveOutputFormatFactory (org.apache.flink.connectors.hive.write.HiveOutputFormatFactory)2 ThreadLocalClassLoaderConfiguration (org.apache.flink.orc.writer.ThreadLocalClassLoaderConfiguration)2 CatalogTable (org.apache.flink.table.catalog.CatalogTable)2 CatalogException (org.apache.flink.table.catalog.exceptions.CatalogException)2 HiveMetastoreClientFactory (org.apache.flink.table.catalog.hive.client.HiveMetastoreClientFactory)2