Search in sources :

Example 1 with NessieConflictException

use of org.projectnessie.error.NessieConflictException in project iceberg by apache.

the class NessieCatalog method dropTable.

@Override
public boolean dropTable(TableIdentifier identifier, boolean purge) {
    reference.checkMutable();
    IcebergTable existingTable = table(identifier);
    if (existingTable == null) {
        return false;
    }
    if (purge) {
        LOG.info("Purging data for table {} was set to true but is ignored", identifier.toString());
    }
    CommitMultipleOperationsBuilder commitBuilderBase = api.commitMultipleOperations().commitMeta(NessieUtil.buildCommitMetadata(String.format("Iceberg delete table %s", identifier), catalogOptions)).operation(Operation.Delete.of(NessieUtil.toKey(identifier)));
    // We try to drop the table. Simple retry after ref update.
    boolean threw = true;
    try {
        Tasks.foreach(commitBuilderBase).retry(5).stopRetryOn(NessieNotFoundException.class).throwFailureWhenFinished().onFailure((o, exception) -> refresh()).run(commitBuilder -> {
            Branch branch = commitBuilder.branch(reference.getAsBranch()).commit();
            reference.updateReference(branch);
        }, BaseNessieClientServerException.class);
        threw = false;
    } catch (NessieConflictException e) {
        LOG.error("Cannot drop table: failed after retry (update ref and retry)", e);
    } catch (NessieNotFoundException e) {
        LOG.error("Cannot drop table: ref is no longer valid.", e);
    } catch (BaseNessieClientServerException e) {
        LOG.error("Cannot drop table: unknown error", e);
    }
    return !threw;
}
Also used : AlreadyExistsException(org.apache.iceberg.exceptions.AlreadyExistsException) HttpClientBuilder(org.projectnessie.client.http.HttpClientBuilder) CatalogUtil(org.apache.iceberg.CatalogUtil) ImmutableMap(org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) LoggerFactory(org.slf4j.LoggerFactory) HadoopFileIO(org.apache.iceberg.hadoop.HadoopFileIO) Function(java.util.function.Function) Reference(org.projectnessie.model.Reference) NessieClientBuilder(org.projectnessie.client.NessieClientBuilder) HttpClientException(org.projectnessie.client.http.HttpClientException) NessieConflictException(org.projectnessie.error.NessieConflictException) CatalogProperties(org.apache.iceberg.CatalogProperties) TableOperations(org.apache.iceberg.TableOperations) NoSuchNamespaceException(org.apache.iceberg.exceptions.NoSuchNamespaceException) Map(java.util.Map) Configuration(org.apache.hadoop.conf.Configuration) BaseMetastoreCatalog(org.apache.iceberg.BaseMetastoreCatalog) NoSuchTableException(org.apache.iceberg.exceptions.NoSuchTableException) Namespace(org.apache.iceberg.catalog.Namespace) Content(org.projectnessie.model.Content) Configurable(org.apache.hadoop.conf.Configurable) SupportsNamespaces(org.apache.iceberg.catalog.SupportsNamespaces) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException) Operation(org.projectnessie.model.Operation) Logger(org.slf4j.Logger) TableIdentifier(org.apache.iceberg.catalog.TableIdentifier) Branch(org.projectnessie.model.Branch) Set(java.util.Set) Collectors(java.util.stream.Collectors) Joiner(org.apache.iceberg.relocated.com.google.common.base.Joiner) NessieApiV1(org.projectnessie.client.api.NessieApiV1) List(java.util.List) Stream(java.util.stream.Stream) NessieConfigConstants(org.projectnessie.client.NessieConfigConstants) IcebergTable(org.projectnessie.model.IcebergTable) Tasks(org.apache.iceberg.util.Tasks) DynMethods(org.apache.iceberg.common.DynMethods) Preconditions(org.apache.iceberg.relocated.com.google.common.base.Preconditions) Tag(org.projectnessie.model.Tag) BaseNessieClientServerException(org.projectnessie.error.BaseNessieClientServerException) ContentKey(org.projectnessie.model.ContentKey) FileIO(org.apache.iceberg.io.FileIO) CommitMultipleOperationsBuilder(org.projectnessie.client.api.CommitMultipleOperationsBuilder) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) VisibleForTesting(org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting) TableReference(org.projectnessie.model.TableReference) CommitMultipleOperationsBuilder(org.projectnessie.client.api.CommitMultipleOperationsBuilder) Branch(org.projectnessie.model.Branch) IcebergTable(org.projectnessie.model.IcebergTable) NessieConflictException(org.projectnessie.error.NessieConflictException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) BaseNessieClientServerException(org.projectnessie.error.BaseNessieClientServerException)

Example 2 with NessieConflictException

use of org.projectnessie.error.NessieConflictException in project iceberg by apache.

the class NessieTableOperations method doCommit.

@Override
protected void doCommit(TableMetadata base, TableMetadata metadata) {
    reference.checkMutable();
    String newMetadataLocation = writeNewMetadata(metadata, currentVersion() + 1);
    boolean delete = true;
    try {
        ImmutableIcebergTable.Builder newTableBuilder = ImmutableIcebergTable.builder();
        if (table != null) {
            newTableBuilder.id(table.getId());
        }
        Snapshot snapshot = metadata.currentSnapshot();
        long snapshotId = snapshot != null ? snapshot.snapshotId() : -1L;
        IcebergTable newTable = newTableBuilder.snapshotId(snapshotId).schemaId(metadata.currentSchemaId()).specId(metadata.defaultSpecId()).sortOrderId(metadata.defaultSortOrderId()).metadataLocation(newMetadataLocation).build();
        LOG.debug("Committing '{}' against '{}': {}", key, reference.getReference(), newTable);
        ImmutableCommitMeta.Builder builder = ImmutableCommitMeta.builder();
        builder.message(buildCommitMsg(base, metadata));
        if (isSnapshotOperation(base, metadata)) {
            builder.putProperties("iceberg.operation", snapshot.operation());
        }
        Branch branch = api.commitMultipleOperations().operation(Operation.Put.of(key, newTable, table)).commitMeta(NessieUtil.catalogOptions(builder, catalogOptions).build()).branch(reference.getAsBranch()).commit();
        reference.updateReference(branch);
        delete = false;
    } catch (NessieConflictException ex) {
        throw new CommitFailedException(ex, "Commit failed: Reference hash is out of date. " + "Update the reference %s and try again", reference.getName());
    } catch (HttpClientException ex) {
        // Intentionally catch all nessie-client-exceptions here and not just the "timeout" variant
        // to catch all kinds of network errors (e.g. connection reset). Network code implementation
        // details and all kinds of network devices can induce unexpected behavior. So better be
        // safe than sorry.
        delete = false;
        throw new CommitStateUnknownException(ex);
    } catch (NessieNotFoundException ex) {
        throw new RuntimeException(String.format("Commit failed: Reference %s no longer exist", reference.getName()), ex);
    } finally {
        if (delete) {
            io().deleteFile(newMetadataLocation);
        }
    }
}
Also used : ImmutableCommitMeta(org.projectnessie.model.ImmutableCommitMeta) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) Snapshot(org.apache.iceberg.Snapshot) HttpClientException(org.projectnessie.client.http.HttpClientException) Branch(org.projectnessie.model.Branch) ImmutableIcebergTable(org.projectnessie.model.ImmutableIcebergTable) IcebergTable(org.projectnessie.model.IcebergTable) NessieConflictException(org.projectnessie.error.NessieConflictException) ImmutableIcebergTable(org.projectnessie.model.ImmutableIcebergTable) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException)

Example 3 with NessieConflictException

use of org.projectnessie.error.NessieConflictException in project nessie by projectnessie.

the class NessieViewOperations method drop.

@Override
public void drop(String viewIdentifier) {
    reference.checkMutable();
    IcebergView existingView = view(toCatalogTableIdentifier(viewIdentifier));
    if (existingView == null) {
        return;
    }
    CommitMultipleOperationsBuilder commitBuilderBase = api.commitMultipleOperations().commitMeta(NessieUtil.buildCommitMetadata(String.format("Iceberg delete view %s", viewIdentifier), catalogOptions)).operation(Operation.Delete.of(NessieUtil.toKey(toCatalogTableIdentifier(viewIdentifier))));
    // We try to drop the view. Simple retry after ref update.
    try {
        Tasks.foreach(commitBuilderBase).retry(5).stopRetryOn(NessieNotFoundException.class).throwFailureWhenFinished().onFailure((o, exception) -> refresh()).run(commitBuilder -> {
            Branch branch = commitBuilder.branch(reference.getAsBranch()).commit();
            reference.updateReference(branch);
        }, BaseNessieClientServerException.class);
    } catch (NessieConflictException e) {
        LOG.error("Cannot drop view: failed after retry (update ref and retry)", e);
    } catch (NessieNotFoundException e) {
        LOG.error("Cannot drop view: ref is no longer valid.", e);
    } catch (BaseNessieClientServerException e) {
        LOG.error("Cannot drop view: unknown error", e);
    }
}
Also used : ImmutableIcebergView(org.projectnessie.model.ImmutableIcebergView) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) LoggerFactory(org.slf4j.LoggerFactory) HttpClientException(org.projectnessie.client.http.HttpClientException) NessieConflictException(org.projectnessie.error.NessieConflictException) ViewVersionMetadata(org.apache.iceberg.view.ViewVersionMetadata) ImmutableCommitMeta(org.projectnessie.model.ImmutableCommitMeta) Map(java.util.Map) NoSuchTableException(org.apache.iceberg.exceptions.NoSuchTableException) Content(org.projectnessie.model.Content) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException) Operation(org.projectnessie.model.Operation) Logger(org.slf4j.Logger) TableIdentifier(org.apache.iceberg.catalog.TableIdentifier) Predicate(java.util.function.Predicate) Branch(org.projectnessie.model.Branch) NessieApiV1(org.projectnessie.client.api.NessieApiV1) IcebergView(org.projectnessie.model.IcebergView) Tasks(org.apache.iceberg.util.Tasks) BaseNessieClientServerException(org.projectnessie.error.BaseNessieClientServerException) ContentKey(org.projectnessie.model.ContentKey) ViewUtils.toCatalogTableIdentifier(org.apache.iceberg.view.ViewUtils.toCatalogTableIdentifier) FileIO(org.apache.iceberg.io.FileIO) CommitMultipleOperationsBuilder(org.projectnessie.client.api.CommitMultipleOperationsBuilder) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) BaseMetastoreViewOperations(org.apache.iceberg.view.BaseMetastoreViewOperations) CommitMultipleOperationsBuilder(org.projectnessie.client.api.CommitMultipleOperationsBuilder) Branch(org.projectnessie.model.Branch) ImmutableIcebergView(org.projectnessie.model.ImmutableIcebergView) IcebergView(org.projectnessie.model.IcebergView) NessieConflictException(org.projectnessie.error.NessieConflictException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) BaseNessieClientServerException(org.projectnessie.error.BaseNessieClientServerException)

Example 4 with NessieConflictException

use of org.projectnessie.error.NessieConflictException in project nessie by projectnessie.

the class NessieViewOperations method commit.

@Override
public void commit(ViewVersionMetadata base, ViewVersionMetadata metadata, Map<String, String> properties) {
    reference.checkMutable();
    String newMetadataLocation = writeNewMetadata(metadata, currentVersion() + 1);
    boolean delete = true;
    try {
        ImmutableIcebergView.Builder viewBuilder = ImmutableIcebergView.builder();
        if (icebergView != null) {
            viewBuilder.id(icebergView.getId());
        }
        IcebergView newView = viewBuilder.metadataLocation(newMetadataLocation).versionId(metadata.currentVersionId()).schemaId(metadata.definition().schema().schemaId()).dialect("TODO: needs to be defined in Iceberg ViewDefinition").sqlText(metadata.definition().sql()).build();
        LOG.debug("Committing '{}' against '{}': {}", key, reference.getReference(), newView);
        ImmutableCommitMeta.Builder builder = ImmutableCommitMeta.builder();
        builder.message(buildCommitMsg(base, metadata) + " " + key.getName());
        Branch branch = api.commitMultipleOperations().operation(Operation.Put.of(key, newView, icebergView)).commitMeta(NessieUtil.catalogOptions(builder, catalogOptions).build()).branch(reference.getAsBranch()).commit();
        reference.updateReference(branch);
        delete = false;
    } catch (NessieConflictException ex) {
        throw new CommitFailedException(ex, "Commit failed: Reference hash is out of date. " + "Update the reference %s and try again", reference.getName());
    } catch (HttpClientException ex) {
        // Intentionally catch all nessie-client-exceptions here and not just the "timeout" variant
        // to catch all kinds of network errors (e.g. connection reset). Network code implementation
        // details and all kinds of network devices can induce unexpected behavior. So better be
        // safe than sorry.
        delete = false;
        throw new CommitStateUnknownException(ex);
    } catch (NessieNotFoundException ex) {
        throw new RuntimeException(String.format("Commit failed: Reference %s no longer exist", reference.getName()), ex);
    } finally {
        if (delete) {
            io().deleteFile(newMetadataLocation);
        }
    }
}
Also used : ImmutableCommitMeta(org.projectnessie.model.ImmutableCommitMeta) ImmutableIcebergView(org.projectnessie.model.ImmutableIcebergView) IcebergView(org.projectnessie.model.IcebergView) CommitStateUnknownException(org.apache.iceberg.exceptions.CommitStateUnknownException) NessieNotFoundException(org.projectnessie.error.NessieNotFoundException) ImmutableIcebergView(org.projectnessie.model.ImmutableIcebergView) HttpClientException(org.projectnessie.client.http.HttpClientException) Branch(org.projectnessie.model.Branch) NessieConflictException(org.projectnessie.error.NessieConflictException) CommitFailedException(org.apache.iceberg.exceptions.CommitFailedException)

Aggregations

CommitFailedException (org.apache.iceberg.exceptions.CommitFailedException)4 CommitStateUnknownException (org.apache.iceberg.exceptions.CommitStateUnknownException)4 HttpClientException (org.projectnessie.client.http.HttpClientException)4 NessieConflictException (org.projectnessie.error.NessieConflictException)4 NessieNotFoundException (org.projectnessie.error.NessieNotFoundException)4 Branch (org.projectnessie.model.Branch)4 Map (java.util.Map)2 TableIdentifier (org.apache.iceberg.catalog.TableIdentifier)2 NoSuchTableException (org.apache.iceberg.exceptions.NoSuchTableException)2 FileIO (org.apache.iceberg.io.FileIO)2 Tasks (org.apache.iceberg.util.Tasks)2 CommitMultipleOperationsBuilder (org.projectnessie.client.api.CommitMultipleOperationsBuilder)2 NessieApiV1 (org.projectnessie.client.api.NessieApiV1)2 BaseNessieClientServerException (org.projectnessie.error.BaseNessieClientServerException)2 IcebergTable (org.projectnessie.model.IcebergTable)2 IcebergView (org.projectnessie.model.IcebergView)2 ImmutableCommitMeta (org.projectnessie.model.ImmutableCommitMeta)2 ImmutableIcebergView (org.projectnessie.model.ImmutableIcebergView)2 List (java.util.List)1 Set (java.util.Set)1