Search in sources :

Example 1 with Transaction

use of org.apache.iceberg.Transaction in project hive by apache.

the class HiveCreateReplaceTableTest method testReplaceTableTxn.

@Test
public void testReplaceTableTxn() {
    catalog.createTable(TABLE_IDENTIFIER, SCHEMA, SPEC, tableLocation, Maps.newHashMap());
    Assert.assertTrue("Table should exist", catalog.tableExists(TABLE_IDENTIFIER));
    Transaction txn = catalog.newReplaceTableTransaction(TABLE_IDENTIFIER, SCHEMA, false);
    txn.commitTransaction();
    Table table = catalog.loadTable(TABLE_IDENTIFIER);
    PartitionSpec v1Expected = PartitionSpec.builderFor(table.schema()).alwaysNull("id", "id").withSpecId(1).build();
    Assert.assertEquals("Table should have a spec with one void field", v1Expected, table.spec());
}
Also used : Table(org.apache.iceberg.Table) Transaction(org.apache.iceberg.Transaction) PartitionSpec(org.apache.iceberg.PartitionSpec) Test(org.junit.Test)

Example 2 with Transaction

use of org.apache.iceberg.Transaction in project hive by apache.

the class HiveCreateReplaceTableTest method testCreateOrReplaceTableTxnTableExists.

@Test
public void testCreateOrReplaceTableTxnTableExists() {
    catalog.createTable(TABLE_IDENTIFIER, SCHEMA, SPEC, tableLocation, Maps.newHashMap());
    Assert.assertTrue("Table should exist", catalog.tableExists(TABLE_IDENTIFIER));
    Transaction txn = catalog.newReplaceTableTransaction(TABLE_IDENTIFIER, SCHEMA, true);
    txn.commitTransaction();
    Table table = catalog.loadTable(TABLE_IDENTIFIER);
    PartitionSpec v1Expected = PartitionSpec.builderFor(table.schema()).alwaysNull("id", "id").withSpecId(1).build();
    Assert.assertEquals("Table should have a spec with one void field", v1Expected, table.spec());
}
Also used : Table(org.apache.iceberg.Table) Transaction(org.apache.iceberg.Transaction) PartitionSpec(org.apache.iceberg.PartitionSpec) Test(org.junit.Test)

Example 3 with Transaction

use of org.apache.iceberg.Transaction in project hive by apache.

the class HiveCreateReplaceTableTest method testCreateTableTxnTableCreatedConcurrently.

@Test
public void testCreateTableTxnTableCreatedConcurrently() {
    Assert.assertFalse("Table should not exist", catalog.tableExists(TABLE_IDENTIFIER));
    Transaction txn = catalog.newCreateTableTransaction(TABLE_IDENTIFIER, SCHEMA, SPEC, tableLocation, Maps.newHashMap());
    // create the table concurrently
    catalog.createTable(TABLE_IDENTIFIER, SCHEMA, SPEC);
    Assert.assertTrue("Table should be created", catalog.tableExists(TABLE_IDENTIFIER));
    AssertHelpers.assertThrows("Create table txn should fail", AlreadyExistsException.class, "Table already exists: hivedb.tbl", txn::commitTransaction);
}
Also used : Transaction(org.apache.iceberg.Transaction) Test(org.junit.Test)

Example 4 with Transaction

use of org.apache.iceberg.Transaction in project hive by apache.

the class HiveCreateReplaceTableTest method testReplaceTableTxnTableModifiedConcurrently.

@Test
public void testReplaceTableTxnTableModifiedConcurrently() {
    Table table = catalog.createTable(TABLE_IDENTIFIER, SCHEMA, SPEC, tableLocation, Maps.newHashMap());
    Assert.assertTrue("Table should exist", catalog.tableExists(TABLE_IDENTIFIER));
    Transaction txn = catalog.newReplaceTableTransaction(TABLE_IDENTIFIER, SCHEMA, SPEC, false);
    // update the table concurrently
    table.updateProperties().set("another-prop", "another-value").commit();
    txn.updateProperties().set("prop", "value").commit();
    txn.commitTransaction();
    // the replace should still succeed
    table = catalog.loadTable(TABLE_IDENTIFIER);
    Assert.assertNull("Table props should be updated", table.properties().get("another-prop"));
    Assert.assertEquals("Table props should match", "value", table.properties().get("prop"));
}
Also used : Table(org.apache.iceberg.Table) Transaction(org.apache.iceberg.Transaction) Test(org.junit.Test)

Example 5 with Transaction

use of org.apache.iceberg.Transaction in project hive by apache.

the class TestHiveCatalog method testCreateTableTxnBuilder.

@Test
public void testCreateTableTxnBuilder() throws Exception {
    Schema schema = new Schema(required(1, "id", Types.IntegerType.get(), "unique ID"), required(2, "data", Types.StringType.get()));
    TableIdentifier tableIdent = TableIdentifier.of(DB_NAME, "tbl");
    String location = temp.newFolder("tbl").toString();
    try {
        Transaction txn = catalog.buildTable(tableIdent, schema).withLocation(location).createTransaction();
        txn.commitTransaction();
        Table table = catalog.loadTable(tableIdent);
        Assert.assertEquals(location, table.location());
        Assert.assertEquals(2, table.schema().columns().size());
        Assert.assertTrue(table.spec().isUnpartitioned());
    } finally {
        catalog.dropTable(tableIdent);
    }
}
Also used : TableIdentifier(org.apache.iceberg.catalog.TableIdentifier) Table(org.apache.iceberg.Table) Transaction(org.apache.iceberg.Transaction) Schema(org.apache.iceberg.Schema) Test(org.junit.Test)

Aggregations

Transaction (org.apache.iceberg.Transaction)16 Test (org.junit.Test)13 Table (org.apache.iceberg.Table)11 DataFile (org.apache.iceberg.DataFile)3 PartitionSpec (org.apache.iceberg.PartitionSpec)3 TableIdentifier (org.apache.iceberg.catalog.TableIdentifier)3 Schema (org.apache.iceberg.Schema)2 Snapshot (org.apache.iceberg.Snapshot)2 IOException (java.io.IOException)1 Lock (java.util.concurrent.locks.Lock)1 ReadWriteLock (java.util.concurrent.locks.ReadWriteLock)1 ReentrantReadWriteLock (java.util.concurrent.locks.ReentrantReadWriteLock)1 AutoCloseableHiveLock (org.apache.gobblin.hive.AutoCloseableHiveLock)1 HiveLock (org.apache.gobblin.hive.HiveLock)1 SchemaRegistryException (org.apache.gobblin.metrics.kafka.SchemaRegistryException)1 Path (org.apache.hadoop.fs.Path)1 AppendFiles (org.apache.iceberg.AppendFiles)1 UpdateProperties (org.apache.iceberg.UpdateProperties)1 GenericAppenderFactory (org.apache.iceberg.data.GenericAppenderFactory)1 GenericRecord (org.apache.iceberg.data.GenericRecord)1