Examples with OutputFile org.apache.iceberg.io.OutputFile used on opensource projects

Example 1 with OutputFile

use of org.apache.iceberg.io.OutputFile in project hive by apache.

the class HiveIcebergOutputCommitter method createFileForCommit.

private static void createFileForCommit(DataFile[] closedFiles, String location, FileIO io) throws IOException {
    OutputFile fileForCommit = io.newOutputFile(location);
    try (ObjectOutputStream oos = new ObjectOutputStream(fileForCommit.createOrOverwrite())) {
        oos.writeObject(closedFiles);
    }
    LOG.debug("Iceberg committed file is created {}", fileForCommit);
}

Example 2 with OutputFile

use of org.apache.iceberg.io.OutputFile in project presto by prestodb.

the class HiveTableOperations method writeNewMetadata.

private String writeNewMetadata(TableMetadata metadata, int newVersion) {
    String newTableMetadataFilePath = newTableMetadataFilePath(metadata, newVersion);
    OutputFile newMetadataLocation = fileIO.newOutputFile(newTableMetadataFilePath);
    // write the new metadata
    TableMetadataParser.write(metadata, newMetadataLocation);
    return newTableMetadataFilePath;
}

Example 3 with OutputFile

use of org.apache.iceberg.io.OutputFile in project drill by apache.

the class ParquetFileWriter method write.

@Override
public File write() {
    Objects.requireNonNull(location, "File create location must be specified");
    Objects.requireNonNull(name, "File name must be specified");
    OutputFile outputFile = table.io().newOutputFile(new Path(location, FileFormat.PARQUET.addExtension(name)).toUri().getPath());
    FileAppender<Record> fileAppender = null;
    try {
        fileAppender = Parquet.write(outputFile).forTable(table).createWriterFunc(GenericParquetWriter::buildWriter).build();
        fileAppender.addAll(records);
        fileAppender.close();
        // metrics are available only when file was written (i.e. close method was executed)
        return new File(outputFile, fileAppender.metrics());
    } catch (IOException | ClassCastException | RuntimeIOException e) {
        if (fileAppender != null) {
            try {
                fileAppender.close();
            } catch (Exception ex) {
            // write has failed anyway, ignore closing exception if any and throw initial one
            }
        }
        throw new IcebergMetastoreException(String.format("Unable to write data into parquet file [%s]", outputFile.location()), e);
    }
}

Example 4 with OutputFile

use of org.apache.iceberg.io.OutputFile in project drill by apache.

the class IcebergQueriesTest method writeAndCommitDataFile.

private static void writeAndCommitDataFile(Table table, String name, FileFormat fileFormat, Iterable<Record> records) throws IOException {
    OutputFile outputFile = table.io().newOutputFile(new Path(table.location(), fileFormat.addExtension(name)).toUri().getPath());
    FileAppender<Record> fileAppender = new GenericAppenderFactory(table.schema()).newAppender(outputFile, fileFormat);
    fileAppender.addAll(records);
    fileAppender.close();
    DataFile dataFile = DataFiles.builder(table.spec()).withInputFile(outputFile.toInputFile()).withMetrics(fileAppender.metrics()).build();
    Transaction transaction = table.newTransaction();
    transaction.newAppend().appendFile(dataFile).commit();
    transaction.commitTransaction();
}