Search in sources :

Example 16 with Transaction

use of org.apache.iceberg.Transaction in project drill by apache.

the class IcebergQueriesTest method writeAndCommitDataFile.

private static void writeAndCommitDataFile(Table table, String name, FileFormat fileFormat, Iterable<Record> records) throws IOException {
    OutputFile outputFile = table.io().newOutputFile(new Path(table.location(), fileFormat.addExtension(name)).toUri().getPath());
    FileAppender<Record> fileAppender = new GenericAppenderFactory(table.schema()).newAppender(outputFile, fileFormat);
    fileAppender.addAll(records);
    fileAppender.close();
    DataFile dataFile = DataFiles.builder(table.spec()).withInputFile(outputFile.toInputFile()).withMetrics(fileAppender.metrics()).build();
    Transaction transaction = table.newTransaction();
    transaction.newAppend().appendFile(dataFile).commit();
    transaction.commitTransaction();
}
Also used : OutputFile(org.apache.iceberg.io.OutputFile) Path(org.apache.hadoop.fs.Path) DataFile(org.apache.iceberg.DataFile) Transaction(org.apache.iceberg.Transaction) GenericRecord(org.apache.iceberg.data.GenericRecord) Record(org.apache.iceberg.data.Record) GenericAppenderFactory(org.apache.iceberg.data.GenericAppenderFactory)

Aggregations

Transaction (org.apache.iceberg.Transaction)16 Test (org.junit.Test)13 Table (org.apache.iceberg.Table)11 DataFile (org.apache.iceberg.DataFile)3 PartitionSpec (org.apache.iceberg.PartitionSpec)3 TableIdentifier (org.apache.iceberg.catalog.TableIdentifier)3 Schema (org.apache.iceberg.Schema)2 Snapshot (org.apache.iceberg.Snapshot)2 IOException (java.io.IOException)1 Lock (java.util.concurrent.locks.Lock)1 ReadWriteLock (java.util.concurrent.locks.ReadWriteLock)1 ReentrantReadWriteLock (java.util.concurrent.locks.ReentrantReadWriteLock)1 AutoCloseableHiveLock (org.apache.gobblin.hive.AutoCloseableHiveLock)1 HiveLock (org.apache.gobblin.hive.HiveLock)1 SchemaRegistryException (org.apache.gobblin.metrics.kafka.SchemaRegistryException)1 Path (org.apache.hadoop.fs.Path)1 AppendFiles (org.apache.iceberg.AppendFiles)1 UpdateProperties (org.apache.iceberg.UpdateProperties)1 GenericAppenderFactory (org.apache.iceberg.data.GenericAppenderFactory)1 GenericRecord (org.apache.iceberg.data.GenericRecord)1