Examples with VisibleForTesting org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting used on opensource projects

Example 1 with VisibleForTesting

use of org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting in project hive by apache.

the class HiveTableOperations method acquireLock.

@VisibleForTesting
long acquireLock() throws UnknownHostException, TException, InterruptedException {
    final LockComponent lockComponent = new LockComponent(LockType.EXCL_WRITE, LockLevel.TABLE, database);
    lockComponent.setTablename(tableName);
    final LockRequest lockRequest = new LockRequest(Lists.newArrayList(lockComponent), System.getProperty("user.name"), InetAddress.getLocalHost().getHostName());
    LockResponse lockResponse = metaClients.run(client -> client.lock(lockRequest));
    AtomicReference<LockState> state = new AtomicReference<>(lockResponse.getState());
    long lockId = lockResponse.getLockid();
    final long start = System.currentTimeMillis();
    long duration = 0;
    boolean timeout = false;
    try {
        if (state.get().equals(LockState.WAITING)) {
            // Retry count is the typical "upper bound of retries" for Tasks.run() function. In fact, the maximum number of
            // attempts the Tasks.run() would try is `retries + 1`. Here, for checking locks, we use timeout as the
            // upper bound of retries. So it is just reasonable to set a large retry count. However, if we set
            // Integer.MAX_VALUE, the above logic of `retries + 1` would overflow into Integer.MIN_VALUE. Hence,
            // the retry is set conservatively as `Integer.MAX_VALUE - 100` so it doesn't hit any boundary issues.
            Tasks.foreach(lockId).retry(Integer.MAX_VALUE - 100).exponentialBackoff(lockCheckMinWaitTime, lockCheckMaxWaitTime, lockAcquireTimeout, 1.5).throwFailureWhenFinished().onlyRetryOn(WaitingForLockException.class).run(id -> {
                try {
                    LockResponse response = metaClients.run(client -> client.checkLock(id));
                    LockState newState = response.getState();
                    state.set(newState);
                    if (newState.equals(LockState.WAITING)) {
                        throw new WaitingForLockException("Waiting for lock.");
                    }
                } catch (InterruptedException e) {
                    // Clear the interrupt status flag
                    Thread.interrupted();
                    LOG.warn("Interrupted while waiting for lock.", e);
                }
            }, TException.class);
        }
    } catch (WaitingForLockException waitingForLockException) {
        timeout = true;
        duration = System.currentTimeMillis() - start;
    } finally {
        if (!state.get().equals(LockState.ACQUIRED)) {
            unlock(Optional.of(lockId));
        }
    }
    // timeout and do not have lock acquired
    if (timeout && !state.get().equals(LockState.ACQUIRED)) {
        throw new CommitFailedException("Timed out after %s ms waiting for lock on %s.%s", duration, database, tableName);
    }
    if (!state.get().equals(LockState.ACQUIRED)) {
        throw new CommitFailedException("Could not acquire the lock on %s.%s, " + "lock request ended in state %s", database, tableName, state);
    }
    return lockId;
}

Example 2 with VisibleForTesting

use of org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting in project hive by apache.

the class HiveTableOperations method persistTable.

@VisibleForTesting
void persistTable(Table hmsTable, boolean updateHiveTable) throws TException, InterruptedException {
    if (updateHiveTable) {
        metaClients.run(client -> {
            EnvironmentContext envContext = new EnvironmentContext(ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE));
            ALTER_TABLE.invoke(client, database, tableName, hmsTable, envContext);
            return null;
        });
    } else {
        metaClients.run(client -> {
            client.createTable(hmsTable);
            return null;
        });
    }
}

Example 3 with VisibleForTesting

use of org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting in project hive by apache.

the class HiveIcebergStorageHandler method overlayTableProperties.

/**
 * Stores the serializable table data in the configuration.
 * Currently the following is handled:
 * <ul>
 *   <li>- Table - in case the table is serializable</li>
 *   <li>- Location</li>
 *   <li>- Schema</li>
 *   <li>- Partition specification</li>
 *   <li>- FileIO for handling table files</li>
 *   <li>- Location provider used for file generation</li>
 *   <li>- Encryption manager for encryption handling</li>
 * </ul>
 * @param configuration The configuration storing the catalog information
 * @param tableDesc The table which we want to store to the configuration
 * @param map The map of the configuration properties which we append with the serialized data
 */
@VisibleForTesting
static void overlayTableProperties(Configuration configuration, TableDesc tableDesc, Map<String, String> map) {
    Properties props = tableDesc.getProperties();
    Table table = IcebergTableUtil.getTable(configuration, props);
    String schemaJson = SchemaParser.toJson(table.schema());
    Maps.fromProperties(props).entrySet().stream().filter(// map overrides tableDesc properties
    entry -> !map.containsKey(entry.getKey())).forEach(entry -> map.put(entry.getKey(), entry.getValue()));
    map.put(InputFormatConfig.TABLE_IDENTIFIER, props.getProperty(Catalogs.NAME));
    map.put(InputFormatConfig.TABLE_LOCATION, table.location());
    map.put(InputFormatConfig.TABLE_SCHEMA, schemaJson);
    props.put(InputFormatConfig.PARTITION_SPEC, PartitionSpecParser.toJson(table.spec()));
    // serialize table object into config
    Table serializableTable = SerializableTable.copyOf(table);
    checkAndSkipIoConfigSerialization(configuration, serializableTable);
    map.put(InputFormatConfig.SERIALIZED_TABLE_PREFIX + tableDesc.getTableName(), SerializationUtil.serializeToBase64(serializableTable));
    // We need to remove this otherwise the job.xml will be invalid as column comments are separated with '\0' and
    // the serialization utils fail to serialize this character
    map.remove("columns.comments");
    // save schema into table props as well to avoid repeatedly hitting the HMS during serde initializations
    // this is an exception to the interface documentation, but it's a safe operation to add this property
    props.put(InputFormatConfig.TABLE_SCHEMA, schemaJson);
}