Search in sources :

Example 1 with HttpClientException

use of org.projectnessie.client.http.HttpClientException in project iceberg by apache.

the class NessieCatalog method renameTable.

@Override
public void renameTable(TableIdentifier from, TableIdentifier toOriginal) {
    reference.checkMutable();
    TableIdentifier to = NessieUtil.removeCatalogName(toOriginal, name());
    IcebergTable existingFromTable = table(from);
    if (existingFromTable == null) {
        throw new NoSuchTableException("table %s doesn't exists", from.name());
    }
    IcebergTable existingToTable = table(to);
    if (existingToTable != null) {
        throw new AlreadyExistsException("table %s already exists", to.name());
    }
    CommitMultipleOperationsBuilder operations = api.commitMultipleOperations().commitMeta(NessieUtil.buildCommitMetadata(String.format("Iceberg rename table from '%s' to '%s'", from, to), catalogOptions)).operation(Operation.Put.of(NessieUtil.toKey(to), existingFromTable, existingFromTable)).operation(Operation.Delete.of(NessieUtil.toKey(from)));
    try {
        Tasks.foreach(operations).retry(5).stopRetryOn(NessieNotFoundException.class).throwFailureWhenFinished().onFailure((o, exception) -> refresh()).run(ops -> {
            Branch branch = ops.branch(reference.getAsBranch()).commit();
            reference.updateReference(branch);
        }, BaseNessieClientServerException.class);
    } catch (NessieNotFoundException e) {
        // and removed by another.
        throw new RuntimeException("Failed to drop table as ref is no longer valid.", e);
    } catch (BaseNessieClientServerException e) {
        throw new CommitFailedException(e, "Failed to rename table: the current reference is not up to date.");
    } catch (HttpClientException ex) {
        // safe than sorry.
        throw new CommitStateUnknownException(ex);
    }
// Intentionally just "throw through" Nessie's HttpClientException here and do not "special case"
// just the "timeout" variant to propagate all kinds of network errors (e.g. connection reset).
// Network code implementation details and all kinds of network devices can induce unexpected
// behavior. So better be safe than sorry.
}
Also used : TableIdentifier(org.apache.iceberg.catalog.TableIdentifier) AlreadyExistsException(org.apache.iceberg.exceptions.AlreadyExistsException) HttpClientBuilder(org.projectnessie.client.http.HttpClientBuilder) CatalogUtil(org.apache.iceberg.CatalogUtil) ImmutableMap(org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) LoggerFactory(org.slf4j.LoggerFactory) HadoopFileIO(org.apache.iceberg.hadoop.HadoopFileIO) Function(java.util.function.Function) Reference(org.projectnessie.model.Reference) NessieClientBuilder(org.projectnessie.client.NessieClientBuilder) HttpClientException(org.projectnessie.client.http.HttpClientException) NessieConflictException(org.projectnessie.error.NessieConflictException) CatalogProperties(org.apache.iceberg.CatalogProperties) TableOperations(org.apache.iceberg.TableOperations) NoSuchNamespaceException(org.apache.iceberg.exceptions.NoSuchNamespaceException) Map(java.util.Map) Configuration(org.apache.hadoop.conf.Configuration) BaseMetastoreCatalog(org.apache.iceberg.BaseMetastoreCatalog) NoSuchTableException(org.apache.iceberg.exceptions.NoSuchTableException) Namespace(org.apache.iceberg.catalog.Namespace) Content(org.projectnessie.model.Content) Configurable(org.apache.hadoop.conf.Configurable) SupportsNamespaces(org.apache.iceberg.catalog.SupportsNamespaces) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException) Operation(org.projectnessie.model.Operation) Logger(org.slf4j.Logger) TableIdentifier(org.apache.iceberg.catalog.TableIdentifier) Branch(org.projectnessie.model.Branch) Set(java.util.Set) Collectors(java.util.stream.Collectors) Joiner(org.apache.iceberg.relocated.com.google.common.base.Joiner) NessieApiV1(org.projectnessie.client.api.NessieApiV1) List(java.util.List) Stream(java.util.stream.Stream) NessieConfigConstants(org.projectnessie.client.NessieConfigConstants) IcebergTable(org.projectnessie.model.IcebergTable) Tasks(org.apache.iceberg.util.Tasks) DynMethods(org.apache.iceberg.common.DynMethods) Preconditions(org.apache.iceberg.relocated.com.google.common.base.Preconditions) Tag(org.projectnessie.model.Tag) BaseNessieClientServerException(org.projectnessie.error.BaseNessieClientServerException) ContentKey(org.projectnessie.model.ContentKey) FileIO(org.apache.iceberg.io.FileIO) CommitMultipleOperationsBuilder(org.projectnessie.client.api.CommitMultipleOperationsBuilder) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) VisibleForTesting(org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting) TableReference(org.projectnessie.model.TableReference) CommitMultipleOperationsBuilder(org.projectnessie.client.api.CommitMultipleOperationsBuilder) AlreadyExistsException(org.apache.iceberg.exceptions.AlreadyExistsException) NoSuchTableException(org.apache.iceberg.exceptions.NoSuchTableException) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) HttpClientException(org.projectnessie.client.http.HttpClientException) Branch(org.projectnessie.model.Branch) IcebergTable(org.projectnessie.model.IcebergTable) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException) BaseNessieClientServerException(org.projectnessie.error.BaseNessieClientServerException)

Example 2 with HttpClientException

use of org.projectnessie.client.http.HttpClientException in project iceberg by apache.

the class NessieTableOperations method doCommit.

@Override
protected void doCommit(TableMetadata base, TableMetadata metadata) {
    reference.checkMutable();
    String newMetadataLocation = writeNewMetadata(metadata, currentVersion() + 1);
    boolean delete = true;
    try {
        ImmutableIcebergTable.Builder newTableBuilder = ImmutableIcebergTable.builder();
        if (table != null) {
            newTableBuilder.id(table.getId());
        }
        Snapshot snapshot = metadata.currentSnapshot();
        long snapshotId = snapshot != null ? snapshot.snapshotId() : -1L;
        IcebergTable newTable = newTableBuilder.snapshotId(snapshotId).schemaId(metadata.currentSchemaId()).specId(metadata.defaultSpecId()).sortOrderId(metadata.defaultSortOrderId()).metadataLocation(newMetadataLocation).build();
        LOG.debug("Committing '{}' against '{}': {}", key, reference.getReference(), newTable);
        ImmutableCommitMeta.Builder builder = ImmutableCommitMeta.builder();
        builder.message(buildCommitMsg(base, metadata));
        if (isSnapshotOperation(base, metadata)) {
            builder.putProperties("iceberg.operation", snapshot.operation());
        }
        Branch branch = api.commitMultipleOperations().operation(Operation.Put.of(key, newTable, table)).commitMeta(NessieUtil.catalogOptions(builder, catalogOptions).build()).branch(reference.getAsBranch()).commit();
        reference.updateReference(branch);
        delete = false;
    } catch (NessieConflictException ex) {
        throw new CommitFailedException(ex, "Commit failed: Reference hash is out of date. " + "Update the reference %s and try again", reference.getName());
    } catch (HttpClientException ex) {
        // Intentionally catch all nessie-client-exceptions here and not just the "timeout" variant
        // to catch all kinds of network errors (e.g. connection reset). Network code implementation
        // details and all kinds of network devices can induce unexpected behavior. So better be
        // safe than sorry.
        delete = false;
        throw new CommitStateUnknownException(ex);
    } catch (NessieNotFoundException ex) {
        throw new RuntimeException(String.format("Commit failed: Reference %s no longer exist", reference.getName()), ex);
    } finally {
        if (delete) {
            io().deleteFile(newMetadataLocation);
        }
    }
}
Also used : ImmutableCommitMeta(org.projectnessie.model.ImmutableCommitMeta) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) Snapshot(org.apache.iceberg.Snapshot) HttpClientException(org.projectnessie.client.http.HttpClientException) Branch(org.projectnessie.model.Branch) ImmutableIcebergTable(org.projectnessie.model.ImmutableIcebergTable) IcebergTable(org.projectnessie.model.IcebergTable) NessieConflictException(org.projectnessie.error.NessieConflictException) ImmutableIcebergTable(org.projectnessie.model.ImmutableIcebergTable) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException)

Example 3 with HttpClientException

use of org.projectnessie.client.http.HttpClientException in project nessie by projectnessie.

the class NessieViewOperations method commit.

@Override
public void commit(ViewVersionMetadata base, ViewVersionMetadata metadata, Map<String, String> properties) {
    reference.checkMutable();
    String newMetadataLocation = writeNewMetadata(metadata, currentVersion() + 1);
    boolean delete = true;
    try {
        ImmutableIcebergView.Builder viewBuilder = ImmutableIcebergView.builder();
        if (icebergView != null) {
            viewBuilder.id(icebergView.getId());
        }
        IcebergView newView = viewBuilder.metadataLocation(newMetadataLocation).versionId(metadata.currentVersionId()).schemaId(metadata.definition().schema().schemaId()).dialect("TODO: needs to be defined in Iceberg ViewDefinition").sqlText(metadata.definition().sql()).build();
        LOG.debug("Committing '{}' against '{}': {}", key, reference.getReference(), newView);
        ImmutableCommitMeta.Builder builder = ImmutableCommitMeta.builder();
        builder.message(buildCommitMsg(base, metadata) + " " + key.getName());
        Branch branch = api.commitMultipleOperations().operation(Operation.Put.of(key, newView, icebergView)).commitMeta(NessieUtil.catalogOptions(builder, catalogOptions).build()).branch(reference.getAsBranch()).commit();
        reference.updateReference(branch);
        delete = false;
    } catch (NessieConflictException ex) {
        throw new CommitFailedException(ex, "Commit failed: Reference hash is out of date. " + "Update the reference %s and try again", reference.getName());
    } catch (HttpClientException ex) {
        // Intentionally catch all nessie-client-exceptions here and not just the "timeout" variant
        // to catch all kinds of network errors (e.g. connection reset). Network code implementation
        // details and all kinds of network devices can induce unexpected behavior. So better be
        // safe than sorry.
        delete = false;
        throw new CommitStateUnknownException(ex);
    } catch (NessieNotFoundException ex) {
        throw new RuntimeException(String.format("Commit failed: Reference %s no longer exist", reference.getName()), ex);
    } finally {
        if (delete) {
            io().deleteFile(newMetadataLocation);
        }
    }
}
Also used : ImmutableCommitMeta(org.projectnessie.model.ImmutableCommitMeta) ImmutableIcebergView(org.projectnessie.model.ImmutableIcebergView) IcebergView(org.projectnessie.model.IcebergView) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) ImmutableIcebergView(org.projectnessie.model.ImmutableIcebergView) HttpClientException(org.projectnessie.client.http.HttpClientException) Branch(org.projectnessie.model.Branch) NessieConflictException(org.projectnessie.error.NessieConflictException) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException)

Aggregations

CommitFailedException (org.apache.iceberg.exceptions.CommitFailedException)3 CommitStateUnknownException (org.apache.iceberg.exceptions.CommitStateUnknownException)3 HttpClientException (org.projectnessie.client.http.HttpClientException)3 NessieConflictException (org.projectnessie.error.NessieConflictException)3 NessieNotFoundException (org.projectnessie.error.NessieNotFoundException)3 Branch (org.projectnessie.model.Branch)3 IcebergTable (org.projectnessie.model.IcebergTable)2 List (java.util.List)1 Map (java.util.Map)1 Set (java.util.Set)1 Function (java.util.function.Function)1 Collectors (java.util.stream.Collectors)1 Stream (java.util.stream.Stream)1 Configurable (org.apache.hadoop.conf.Configurable)1 Configuration (org.apache.hadoop.conf.Configuration)1 BaseMetastoreCatalog (org.apache.iceberg.BaseMetastoreCatalog)1 CatalogProperties (org.apache.iceberg.CatalogProperties)1 CatalogUtil (org.apache.iceberg.CatalogUtil)1 Snapshot (org.apache.iceberg.Snapshot)1 TableOperations (org.apache.iceberg.TableOperations)1