Search in sources :

Example 11 with BranchName

use of org.projectnessie.versioned.BranchName in project nessie by projectnessie.

the class AbstractMergeTransplant method transplant.

@ParameterizedTest
@ValueSource(ints = { 3, 10, DEFAULT_KEY_LIST_DISTANCE, DEFAULT_KEY_LIST_DISTANCE + 1, 100 })
void transplant(int numCommits) throws Exception {
    AtomicInteger unifier = new AtomicInteger();
    Function<ByteString, ByteString> metadataUpdater = commitMeta -> ByteString.copyFromUtf8(commitMeta.toStringUtf8() + " transplanted " + unifier.getAndIncrement());
    Hash[] commits = mergeTransplant(numCommits, (target, expectedHead, branch, commitHashes, i) -> databaseAdapter.transplant(target, expectedHead, Arrays.asList(commitHashes).subList(0, i + 1), metadataUpdater));
    BranchName conflict = BranchName.of("conflict");
    // no conflict, when transplanting the commits from against the current HEAD of the
    // conflict-branch
    Hash noConflictHead = databaseAdapter.hashOnReference(conflict, Optional.empty());
    Hash transplanted = databaseAdapter.transplant(conflict, Optional.of(noConflictHead), Arrays.asList(commits), metadataUpdater);
    int offset = unifier.get();
    try (Stream<CommitLogEntry> log = databaseAdapter.commitLog(transplanted).limit(commits.length)) {
        AtomicInteger testOffset = new AtomicInteger(offset);
        assertThat(log.map(CommitLogEntry::getMetadata).map(ByteString::toStringUtf8)).containsExactlyElementsOf(IntStream.range(0, commits.length).map(i -> commits.length - i - 1).mapToObj(i -> "commit " + i + " transplanted " + testOffset.decrementAndGet()).collect(Collectors.toList()));
    }
    // again, no conflict (same as above, just again)
    transplanted = databaseAdapter.transplant(conflict, Optional.empty(), Arrays.asList(commits), metadataUpdater);
    offset = unifier.get();
    try (Stream<CommitLogEntry> log = databaseAdapter.commitLog(transplanted).limit(commits.length)) {
        AtomicInteger testOffset = new AtomicInteger(offset);
        assertThat(log.map(CommitLogEntry::getMetadata).map(ByteString::toStringUtf8)).containsExactlyElementsOf(IntStream.range(0, commits.length).map(i -> commits.length - i - 1).mapToObj(i -> "commit " + i + " transplanted " + testOffset.decrementAndGet()).collect(Collectors.toList()));
    }
    assertThatThrownBy(() -> databaseAdapter.transplant(conflict, Optional.empty(), Collections.emptyList(), Function.identity())).isInstanceOf(IllegalArgumentException.class).hasMessage("No hashes to transplant given.");
}
Also used : IntStream(java.util.stream.IntStream) ValueSource(org.junit.jupiter.params.provider.ValueSource) Arrays(java.util.Arrays) DEFAULT_KEY_LIST_DISTANCE(org.projectnessie.versioned.persist.adapter.DatabaseAdapterConfig.DEFAULT_KEY_LIST_DISTANCE) Hash(org.projectnessie.versioned.Hash) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) Key(org.projectnessie.versioned.Key) Function(java.util.function.Function) Collectors(java.util.stream.Collectors) KeyWithBytes(org.projectnessie.versioned.persist.adapter.KeyWithBytes) ByteString(com.google.protobuf.ByteString) BranchName(org.projectnessie.versioned.BranchName) ReferenceConflictException(org.projectnessie.versioned.ReferenceConflictException) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) Stream(java.util.stream.Stream) Assertions.assertThatThrownBy(org.assertj.core.api.Assertions.assertThatThrownBy) ImmutableCommitAttempt(org.projectnessie.versioned.persist.adapter.ImmutableCommitAttempt) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) CommitLogEntry(org.projectnessie.versioned.persist.adapter.CommitLogEntry) Optional(java.util.Optional) Collections(java.util.Collections) ContentId(org.projectnessie.versioned.persist.adapter.ContentId) DatabaseAdapter(org.projectnessie.versioned.persist.adapter.DatabaseAdapter) CommitLogEntry(org.projectnessie.versioned.persist.adapter.CommitLogEntry) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) ByteString(com.google.protobuf.ByteString) BranchName(org.projectnessie.versioned.BranchName) Hash(org.projectnessie.versioned.Hash) ValueSource(org.junit.jupiter.params.provider.ValueSource) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 12 with BranchName

use of org.projectnessie.versioned.BranchName in project nessie by projectnessie.

the class AbstractDatabaseAdapter method transplantAttempt.

/**
 * Logic implementation of a transplant-attempt.
 *
 * @param ctx technical operation context
 * @param targetBranch target reference with expected HEAD
 * @param expectedHead if present, {@code targetBranch}'s current HEAD must be equal to this value
 * @param targetHead current HEAD of {@code targetBranch}
 * @param sequenceToTransplant sequential list of commits to transplant from {@code source}
 * @param branchCommits consumer for the individual commits to merge
 * @param newKeyLists consumer for optimistically written {@link KeyListEntity}s
 * @param rewriteMetadata function to rewrite the commit-metadata for copied commits
 * @return hash of the last commit-log-entry written to {@code targetBranch}
 */
protected Hash transplantAttempt(OP_CONTEXT ctx, long timeInMicros, BranchName targetBranch, Optional<Hash> expectedHead, Hash targetHead, List<Hash> sequenceToTransplant, Consumer<Hash> branchCommits, Consumer<Hash> newKeyLists, Function<ByteString, ByteString> rewriteMetadata) throws ReferenceNotFoundException, ReferenceConflictException {
    if (sequenceToTransplant.isEmpty()) {
        throw new IllegalArgumentException("No hashes to transplant given.");
    }
    // 1. ensure 'expectedHash' is a parent of HEAD-of-'targetBranch' & collect keys
    List<CommitLogEntry> targetEntriesReverseChronological = new ArrayList<>();
    hashOnRef(ctx, targetHead, targetBranch, expectedHead, targetEntriesReverseChronological::add);
    // Exclude the expected-hash on the target-branch from key-collisions check
    if (!targetEntriesReverseChronological.isEmpty() && expectedHead.isPresent() && targetEntriesReverseChronological.get(0).getHash().equals(expectedHead.get())) {
        targetEntriesReverseChronological.remove(0);
    }
    Collections.reverse(targetEntriesReverseChronological);
    // 2. Collect modified keys.
    Set<Key> keysTouchedOnTarget = collectModifiedKeys(targetEntriesReverseChronological);
    // 4. ensure 'sequenceToTransplant' is sequential
    int[] index = new int[] { sequenceToTransplant.size() - 1 };
    Hash lastHash = sequenceToTransplant.get(sequenceToTransplant.size() - 1);
    List<CommitLogEntry> commitsToTransplantChronological = takeUntilExcludeLast(readCommitLogStream(ctx, lastHash), e -> {
        int i = index[0]--;
        if (i == -1) {
            return true;
        }
        if (!e.getHash().equals(sequenceToTransplant.get(i))) {
            throw new IllegalArgumentException("Sequence of hashes is not contiguous.");
        }
        return false;
    }).collect(Collectors.toList());
    // 5. check for key-collisions
    checkForKeyCollisions(ctx, targetHead, keysTouchedOnTarget, commitsToTransplantChronological);
    // (no need to verify the global states during a transplant)
    // 6. re-apply commits in 'sequenceToTransplant' onto 'targetBranch'
    targetHead = copyCommits(ctx, timeInMicros, targetHead, commitsToTransplantChronological, newKeyLists, rewriteMetadata);
    // 7. Write commits
    commitsToTransplantChronological.stream().map(CommitLogEntry::getHash).forEach(branchCommits);
    writeMultipleCommits(ctx, commitsToTransplantChronological);
    return targetHead;
}
Also used : Spliterators(java.util.Spliterators) BiFunction(java.util.function.BiFunction) DatabaseAdapterUtil.referenceNotFound(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.referenceNotFound) KeyListEntity(org.projectnessie.versioned.persist.adapter.KeyListEntity) DatabaseAdapterUtil.takeUntilExcludeLast(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.takeUntilExcludeLast) GetNamedRefsParams(org.projectnessie.versioned.GetNamedRefsParams) DatabaseAdapterMetrics.tryLoopFinished(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterMetrics.tryLoopFinished) CommitAttempt(org.projectnessie.versioned.persist.adapter.CommitAttempt) Map(java.util.Map) ContentVariant(org.projectnessie.versioned.persist.adapter.ContentVariant) DatabaseAdapter(org.projectnessie.versioned.persist.adapter.DatabaseAdapter) DatabaseAdapterUtil.hashKey(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.hashKey) NamedRef(org.projectnessie.versioned.NamedRef) Predicate(java.util.function.Predicate) Collection(java.util.Collection) Set(java.util.Set) DatabaseAdapterUtil.takeUntilIncludeLast(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.takeUntilIncludeLast) Instant(java.time.Instant) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) DatabaseAdapterUtil.newHasher(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.newHasher) ByteString(com.google.protobuf.ByteString) Objects(java.util.Objects) BranchName(org.projectnessie.versioned.BranchName) ReferenceConflictException(org.projectnessie.versioned.ReferenceConflictException) List(java.util.List) KeyList(org.projectnessie.versioned.persist.adapter.KeyList) Stream(java.util.stream.Stream) Difference(org.projectnessie.versioned.persist.adapter.Difference) Entry(java.util.Map.Entry) DatabaseAdapterConfig(org.projectnessie.versioned.persist.adapter.DatabaseAdapterConfig) Optional(java.util.Optional) Spliterator(java.util.Spliterator) IntStream(java.util.stream.IntStream) AbstractSpliterator(java.util.Spliterators.AbstractSpliterator) NANOSECONDS(java.util.concurrent.TimeUnit.NANOSECONDS) RefLogNotFoundException(org.projectnessie.versioned.RefLogNotFoundException) TagName(org.projectnessie.versioned.TagName) KeyListEntry(org.projectnessie.versioned.persist.adapter.KeyListEntry) HashMap(java.util.HashMap) Callable(java.util.concurrent.Callable) Function(java.util.function.Function) UnsafeByteOperations(com.google.protobuf.UnsafeByteOperations) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) KeyFilterPredicate(org.projectnessie.versioned.persist.adapter.KeyFilterPredicate) RefLog(org.projectnessie.versioned.persist.adapter.RefLog) CommitLogEntry(org.projectnessie.versioned.persist.adapter.CommitLogEntry) StreamSupport(java.util.stream.StreamSupport) Hasher(com.google.common.hash.Hasher) Nonnull(javax.annotation.Nonnull) ContentVariantSupplier(org.projectnessie.versioned.persist.adapter.ContentVariantSupplier) Iterator(java.util.Iterator) Hash(org.projectnessie.versioned.Hash) Traced.trace(org.projectnessie.versioned.persist.adapter.spi.Traced.trace) ALLOW_ALL(org.projectnessie.versioned.persist.adapter.KeyFilterPredicate.ALLOW_ALL) DatabaseAdapterUtil.hashNotFound(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.hashNotFound) Key(org.projectnessie.versioned.Key) ReferenceNotFoundException(org.projectnessie.versioned.ReferenceNotFoundException) KeyWithBytes(org.projectnessie.versioned.persist.adapter.KeyWithBytes) DatabaseAdapterUtil.randomHash(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.randomHash) RetrieveOptions(org.projectnessie.versioned.GetNamedRefsParams.RetrieveOptions) CommitsAheadBehind(org.projectnessie.versioned.ReferenceInfo.CommitsAheadBehind) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) ContentAndState(org.projectnessie.versioned.persist.adapter.ContentAndState) ImmutableCommitLogEntry(org.projectnessie.versioned.persist.adapter.ImmutableCommitLogEntry) ReferenceInfo(org.projectnessie.versioned.ReferenceInfo) Preconditions(com.google.common.base.Preconditions) Diff(org.projectnessie.versioned.Diff) ImmutableReferenceInfo(org.projectnessie.versioned.ImmutableReferenceInfo) Collections(java.util.Collections) ContentId(org.projectnessie.versioned.persist.adapter.ContentId) ImmutableKeyList(org.projectnessie.versioned.persist.adapter.ImmutableKeyList) CommitLogEntry(org.projectnessie.versioned.persist.adapter.CommitLogEntry) ImmutableCommitLogEntry(org.projectnessie.versioned.persist.adapter.ImmutableCommitLogEntry) ArrayList(java.util.ArrayList) Hash(org.projectnessie.versioned.Hash) DatabaseAdapterUtil.randomHash(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.randomHash) DatabaseAdapterUtil.hashKey(org.projectnessie.versioned.persist.adapter.spi.DatabaseAdapterUtil.hashKey) Key(org.projectnessie.versioned.Key)

Example 13 with BranchName

use of org.projectnessie.versioned.BranchName in project nessie by projectnessie.

the class NamespaceApiImpl method deleteNamespace.

@Override
public void deleteNamespace(NamespaceParams params) throws NessieReferenceNotFoundException, NessieNamespaceNotEmptyException, NessieNamespaceNotFoundException {
    BranchName branch = branchFromRefName(params.getRefName());
    try {
        Namespace namespace = getNamespace(params, branch);
        Delete delete = Delete.of(ContentKey.of(namespace.getElements()));
        Callable<Void> validator = () -> {
            try (Stream<WithType<Key, Type>> keys = getStore().getKeys(branch)) {
                if (keys.anyMatch(k -> Namespace.of(k.getValue().getElements()).name().startsWith(params.getNamespace().name()) && k.getType() != Type.NAMESPACE)) {
                    throw namespaceNotEmptyException(params);
                }
            }
            return null;
        };
        commit(branch, "delete namespace " + namespace.name(), TreeApiImpl.toOp(delete), validator);
    } catch (ReferenceNotFoundException | ReferenceConflictException e) {
        throw new NessieReferenceNotFoundException(e.getMessage(), e);
    }
}
Also used : Delete(org.projectnessie.model.Operation.Delete) NessieNamespaceNotEmptyException(org.projectnessie.error.NessieNamespaceNotEmptyException) ImmutableGetNamespacesResponse(org.projectnessie.model.ImmutableGetNamespacesResponse) Put(org.projectnessie.model.Operation.Put) ServerConfig(org.projectnessie.services.config.ServerConfig) NessieNamespaceAlreadyExistsException(org.projectnessie.error.NessieNamespaceAlreadyExistsException) Authorizer(org.projectnessie.services.authz.Authorizer) NessieReferenceNotFoundException(org.projectnessie.error.NessieReferenceNotFoundException) NamespaceApi(org.projectnessie.api.NamespaceApi) Callable(java.util.concurrent.Callable) GetNamespacesResponse(org.projectnessie.model.GetNamespacesResponse) MultipleNamespacesParams(org.projectnessie.api.params.MultipleNamespacesParams) WithType(org.projectnessie.versioned.WithType) VersionStore(org.projectnessie.versioned.VersionStore) Type(org.projectnessie.model.Content.Type) Map(java.util.Map) Content(org.projectnessie.model.Content) CommitMeta(org.projectnessie.model.CommitMeta) NessieNamespaceNotFoundException(org.projectnessie.error.NessieNamespaceNotFoundException) Nullable(javax.annotation.Nullable) NamespaceParams(org.projectnessie.api.params.NamespaceParams) Set(java.util.Set) Operation(org.projectnessie.versioned.Operation) Key(org.projectnessie.versioned.Key) Collectors(java.util.stream.Collectors) ReferenceNotFoundException(org.projectnessie.versioned.ReferenceNotFoundException) Sets(com.google.common.collect.Sets) BranchName(org.projectnessie.versioned.BranchName) ReferenceConflictException(org.projectnessie.versioned.ReferenceConflictException) List(java.util.List) Principal(java.security.Principal) Stream(java.util.stream.Stream) Delete(org.projectnessie.model.Operation.Delete) Namespace(org.projectnessie.model.Namespace) Optional(java.util.Optional) ContentKey(org.projectnessie.model.ContentKey) Collections(java.util.Collections) ReferenceConflictException(org.projectnessie.versioned.ReferenceConflictException) Namespace(org.projectnessie.model.Namespace) WithType(org.projectnessie.versioned.WithType) Type(org.projectnessie.model.Content.Type) NessieReferenceNotFoundException(org.projectnessie.error.NessieReferenceNotFoundException) ReferenceNotFoundException(org.projectnessie.versioned.ReferenceNotFoundException) NessieReferenceNotFoundException(org.projectnessie.error.NessieReferenceNotFoundException) Stream(java.util.stream.Stream) BranchName(org.projectnessie.versioned.BranchName) Key(org.projectnessie.versioned.Key) ContentKey(org.projectnessie.model.ContentKey)

Example 14 with BranchName

use of org.projectnessie.versioned.BranchName in project nessie by projectnessie.

the class NamespaceApiImpl method getNamespaces.

@Override
public GetNamespacesResponse getNamespaces(MultipleNamespacesParams params) throws NessieReferenceNotFoundException {
    BranchName branch = branchFromRefName(params.getRefName());
    try {
        Set<Namespace> allNamespaces = Sets.newHashSet(getExplicitlyCreatedNamespaces(params, branch));
        List<Namespace> implicitlyCreatedNamespaces = getImplicitlyCreatedNamespaces(params, branch);
        allNamespaces.addAll(implicitlyCreatedNamespaces);
        return ImmutableGetNamespacesResponse.builder().addAllNamespaces(allNamespaces).build();
    } catch (ReferenceNotFoundException e) {
        throw refNotFoundException(e);
    }
}
Also used : NessieReferenceNotFoundException(org.projectnessie.error.NessieReferenceNotFoundException) ReferenceNotFoundException(org.projectnessie.versioned.ReferenceNotFoundException) BranchName(org.projectnessie.versioned.BranchName) Namespace(org.projectnessie.model.Namespace)

Example 15 with BranchName

use of org.projectnessie.versioned.BranchName in project nessie by projectnessie.

the class NamespaceApiImpl method createNamespace.

@Override
public Namespace createNamespace(NamespaceParams params) throws NessieNamespaceAlreadyExistsException, NessieReferenceNotFoundException {
    try {
        BranchName branch = branchFromRefName(params.getRefName());
        Callable<Void> validator = () -> {
            if (getExplicitlyCreatedNamespace(params, branch).isPresent()) {
                throw namespaceAlreadyExistsException(params);
            }
            if (getImplicitlyCreatedNamespace(params, branch).isPresent()) {
                throw namespaceAlreadyExistsException(params);
            }
            return null;
        };
        Namespace namespace = params.getNamespace();
        Put put = Put.of(ContentKey.of(namespace.getElements()), namespace);
        commit(branch, "create namespace " + namespace.name(), TreeApiImpl.toOp(put), validator);
        return namespace;
    } catch (ReferenceNotFoundException | ReferenceConflictException e) {
        throw new NessieReferenceNotFoundException(e.getMessage(), e);
    }
}
Also used : NessieReferenceNotFoundException(org.projectnessie.error.NessieReferenceNotFoundException) ReferenceNotFoundException(org.projectnessie.versioned.ReferenceNotFoundException) NessieReferenceNotFoundException(org.projectnessie.error.NessieReferenceNotFoundException) ReferenceConflictException(org.projectnessie.versioned.ReferenceConflictException) BranchName(org.projectnessie.versioned.BranchName) Namespace(org.projectnessie.model.Namespace) Put(org.projectnessie.model.Operation.Put)

Aggregations

BranchName (org.projectnessie.versioned.BranchName)72 Test (org.junit.jupiter.api.Test)51 Hash (org.projectnessie.versioned.Hash)45 Key (org.projectnessie.versioned.Key)23 ReferenceConflictException (org.projectnessie.versioned.ReferenceConflictException)16 Stream (java.util.stream.Stream)14 ByteString (com.google.protobuf.ByteString)13 Optional (java.util.Optional)13 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)13 Collectors (java.util.stream.Collectors)12 IntStream (java.util.stream.IntStream)11 Assertions.assertThat (org.assertj.core.api.Assertions.assertThat)11 ImmutableCommitAttempt (org.projectnessie.versioned.persist.adapter.ImmutableCommitAttempt)11 List (java.util.List)10 ContentId (org.projectnessie.versioned.persist.adapter.ContentId)10 DatabaseAdapter (org.projectnessie.versioned.persist.adapter.DatabaseAdapter)10 KeyWithBytes (org.projectnessie.versioned.persist.adapter.KeyWithBytes)10 Collections (java.util.Collections)9 ReferenceNotFoundException (org.projectnessie.versioned.ReferenceNotFoundException)9 TagName (org.projectnessie.versioned.TagName)9