Search in sources :

Example 1 with TxnOp

use of org.apache.bookkeeper.api.kv.op.TxnOp in project bookkeeper by apache.

the class RootRangeStoreImpl method executeCreateNamespaceTxn.

private CompletableFuture<CreateNamespaceResponse> executeCreateNamespaceTxn(long currentNsId, long currentNsIdRev, CreateNamespaceRequest request) {
    long namespaceId = currentNsId + 1;
    String nsName = request.getName();
    NamespaceMetadata metadata = NamespaceMetadata.newBuilder().setProps(NamespaceProperties.newBuilder().setNamespaceId(namespaceId).setNamespaceName(nsName).setDefaultStreamConf(request.getColConf().getDefaultStreamConf())).build();
    byte[] nsNameKey = getNamespaceNameKey(nsName);
    byte[] nsNameVal = Bytes.toBytes(namespaceId);
    byte[] nsIdKey = getNamespaceIdKey(namespaceId);
    byte[] nsIdVal = metadata.toByteArray();
    TxnOp<byte[], byte[]> txn = store.newTxn().If(store.newCompareValue(CompareResult.EQUAL, nsNameKey, null), currentNsIdRev < 0 ? store.newCompareValue(CompareResult.EQUAL, NS_ID_KEY, null) : store.newCompareModRevision(CompareResult.EQUAL, NS_ID_KEY, currentNsIdRev)).Then(store.newPut(nsNameKey, nsNameVal), store.newPut(nsIdKey, nsIdVal), store.newPut(NS_ID_KEY, Bytes.toBytes(namespaceId))).build();
    return store.txn(txn).thenApply(txnResult -> {
        try {
            CreateNamespaceResponse.Builder respBuilder = CreateNamespaceResponse.newBuilder();
            if (txnResult.isSuccess()) {
                respBuilder.setCode(StatusCode.SUCCESS);
                respBuilder.setColProps(metadata.getProps());
            } else {
                // TODO: differentiate the error code
                respBuilder.setCode(StatusCode.INTERNAL_SERVER_ERROR);
            }
            return respBuilder.build();
        } finally {
            txnResult.close();
        }
    }).whenComplete((resp, cause) -> txn.close());
}
Also used : ProtoUtils.validateStreamName(org.apache.bookkeeper.stream.protocol.util.ProtoUtils.validateStreamName) GetStreamRequest(org.apache.bookkeeper.stream.proto.storage.GetStreamRequest) Options(org.apache.bookkeeper.api.kv.options.Options) DeleteNamespaceRequest(org.apache.bookkeeper.stream.proto.storage.DeleteNamespaceRequest) CompletableFuture(java.util.concurrent.CompletableFuture) CreateStreamRequest(org.apache.bookkeeper.stream.proto.storage.CreateStreamRequest) GetStreamResponse(org.apache.bookkeeper.stream.proto.storage.GetStreamResponse) DeleteStreamRequest(org.apache.bookkeeper.stream.proto.storage.DeleteStreamRequest) UTF_8(com.google.common.base.Charsets.UTF_8) MIN_DATA_STREAM_ID(org.apache.bookkeeper.stream.protocol.ProtocolConstants.MIN_DATA_STREAM_ID) StreamName(org.apache.bookkeeper.stream.proto.StreamName) CreateStreamResponse(org.apache.bookkeeper.stream.proto.storage.CreateStreamResponse) DeleteStreamResponse(org.apache.bookkeeper.stream.proto.storage.DeleteStreamResponse) MVCCAsyncStore(org.apache.bookkeeper.statelib.api.mvcc.MVCCAsyncStore) TxnOp(org.apache.bookkeeper.api.kv.op.TxnOp) NamespaceProperties(org.apache.bookkeeper.stream.proto.NamespaceProperties) NamespaceMetadata(org.apache.bookkeeper.stream.proto.NamespaceMetadata) DeleteNamespaceResponse(org.apache.bookkeeper.stream.proto.storage.DeleteNamespaceResponse) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) KeyValue(org.apache.bookkeeper.api.kv.result.KeyValue) URI(java.net.URI) StatusCode(org.apache.bookkeeper.stream.proto.storage.StatusCode) InvalidProtocolBufferException(com.google.protobuf.InvalidProtocolBufferException) StorageContainerPlacementPolicy(org.apache.bookkeeper.stream.protocol.util.StorageContainerPlacementPolicy) CreateNamespaceResponse(org.apache.bookkeeper.stream.proto.storage.CreateNamespaceResponse) RangeOp(org.apache.bookkeeper.api.kv.op.RangeOp) RootRangeStore(org.apache.bookkeeper.stream.storage.api.metadata.RootRangeStore) StreamConfiguration(org.apache.bookkeeper.stream.proto.StreamConfiguration) Bytes(org.apache.bookkeeper.common.util.Bytes) FutureUtils(org.apache.bookkeeper.common.concurrent.FutureUtils) Slf4j(lombok.extern.slf4j.Slf4j) CreateNamespaceRequest(org.apache.bookkeeper.stream.proto.storage.CreateNamespaceRequest) StringUtils.isBlank(org.apache.commons.lang3.StringUtils.isBlank) StreamProperties(org.apache.bookkeeper.stream.proto.StreamProperties) CompareResult(org.apache.bookkeeper.api.kv.op.CompareResult) ProtoUtils.validateNamespaceName(org.apache.bookkeeper.stream.protocol.util.ProtoUtils.validateNamespaceName) GetNamespaceRequest(org.apache.bookkeeper.stream.proto.storage.GetNamespaceRequest) GetNamespaceResponse(org.apache.bookkeeper.stream.proto.storage.GetNamespaceResponse) NamespaceMetadata(org.apache.bookkeeper.stream.proto.NamespaceMetadata)

Example 2 with TxnOp

use of org.apache.bookkeeper.api.kv.op.TxnOp in project bookkeeper by apache.

the class RootRangeStoreImpl method executeCreateStreamTxn.

private CompletableFuture<CreateStreamResponse> executeCreateStreamTxn(long nsId, String streamName, StreamConfiguration streamConf, long currentStreamId, long currentStreamIdRev) {
    long streamId;
    if (currentStreamId < 0) {
        streamId = MIN_DATA_STREAM_ID;
    } else {
        streamId = currentStreamId + 1;
    }
    long scId = placementPolicy.placeStreamRange(streamId, 0L);
    StreamConfiguration newStreamConf = streamConf;
    // no backend service url is provided, use the default service url
    if (isBlank(streamConf.getBackendServiceUrl())) {
        newStreamConf = StreamConfiguration.newBuilder(streamConf).setBackendServiceUrl(defaultServiceUri.toString()).build();
    }
    StreamProperties streamProps = StreamProperties.newBuilder().setStreamId(streamId).setStreamName(streamName).setStorageContainerId(scId).setStreamConf(newStreamConf).build();
    byte[] nsIdKey = getNamespaceIdKey(nsId);
    byte[] streamNameKey = getStreamNameKey(nsId, streamName);
    byte[] streamNameVal = Bytes.toBytes(streamId);
    byte[] streamIdKey = getStreamIdKey(nsId, streamId);
    byte[] streamIdVal = streamProps.toByteArray();
    TxnOp<byte[], byte[]> txn = store.newTxn().If(store.newCompareValue(CompareResult.NOT_EQUAL, nsIdKey, null), currentStreamIdRev < 0 ? store.newCompareValue(CompareResult.EQUAL, STREAM_ID_KEY, null) : store.newCompareModRevision(CompareResult.EQUAL, STREAM_ID_KEY, currentStreamIdRev), store.newCompareValue(CompareResult.EQUAL, streamNameKey, null)).Then(store.newPut(streamNameKey, streamNameVal), store.newPut(streamIdKey, streamIdVal), store.newPut(STREAM_ID_KEY, Bytes.toBytes(streamId))).build();
    return store.txn(txn).thenApply(txnResult -> {
        try {
            CreateStreamResponse.Builder respBuilder = CreateStreamResponse.newBuilder();
            if (txnResult.isSuccess()) {
                respBuilder.setCode(StatusCode.SUCCESS);
                respBuilder.setStreamProps(streamProps);
            } else {
                // TODO: differentiate the error codes
                respBuilder.setCode(StatusCode.INTERNAL_SERVER_ERROR);
            }
            return respBuilder.build();
        } finally {
            txnResult.close();
            txn.close();
        }
    }).exceptionally(cause -> {
        txn.close();
        return CreateStreamResponse.newBuilder().setCode(StatusCode.INTERNAL_SERVER_ERROR).build();
    });
}
Also used : ProtoUtils.validateStreamName(org.apache.bookkeeper.stream.protocol.util.ProtoUtils.validateStreamName) GetStreamRequest(org.apache.bookkeeper.stream.proto.storage.GetStreamRequest) Options(org.apache.bookkeeper.api.kv.options.Options) DeleteNamespaceRequest(org.apache.bookkeeper.stream.proto.storage.DeleteNamespaceRequest) CompletableFuture(java.util.concurrent.CompletableFuture) CreateStreamRequest(org.apache.bookkeeper.stream.proto.storage.CreateStreamRequest) GetStreamResponse(org.apache.bookkeeper.stream.proto.storage.GetStreamResponse) DeleteStreamRequest(org.apache.bookkeeper.stream.proto.storage.DeleteStreamRequest) UTF_8(com.google.common.base.Charsets.UTF_8) MIN_DATA_STREAM_ID(org.apache.bookkeeper.stream.protocol.ProtocolConstants.MIN_DATA_STREAM_ID) StreamName(org.apache.bookkeeper.stream.proto.StreamName) CreateStreamResponse(org.apache.bookkeeper.stream.proto.storage.CreateStreamResponse) DeleteStreamResponse(org.apache.bookkeeper.stream.proto.storage.DeleteStreamResponse) MVCCAsyncStore(org.apache.bookkeeper.statelib.api.mvcc.MVCCAsyncStore) TxnOp(org.apache.bookkeeper.api.kv.op.TxnOp) NamespaceProperties(org.apache.bookkeeper.stream.proto.NamespaceProperties) NamespaceMetadata(org.apache.bookkeeper.stream.proto.NamespaceMetadata) DeleteNamespaceResponse(org.apache.bookkeeper.stream.proto.storage.DeleteNamespaceResponse) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) KeyValue(org.apache.bookkeeper.api.kv.result.KeyValue) URI(java.net.URI) StatusCode(org.apache.bookkeeper.stream.proto.storage.StatusCode) InvalidProtocolBufferException(com.google.protobuf.InvalidProtocolBufferException) StorageContainerPlacementPolicy(org.apache.bookkeeper.stream.protocol.util.StorageContainerPlacementPolicy) CreateNamespaceResponse(org.apache.bookkeeper.stream.proto.storage.CreateNamespaceResponse) RangeOp(org.apache.bookkeeper.api.kv.op.RangeOp) RootRangeStore(org.apache.bookkeeper.stream.storage.api.metadata.RootRangeStore) StreamConfiguration(org.apache.bookkeeper.stream.proto.StreamConfiguration) Bytes(org.apache.bookkeeper.common.util.Bytes) FutureUtils(org.apache.bookkeeper.common.concurrent.FutureUtils) Slf4j(lombok.extern.slf4j.Slf4j) CreateNamespaceRequest(org.apache.bookkeeper.stream.proto.storage.CreateNamespaceRequest) StringUtils.isBlank(org.apache.commons.lang3.StringUtils.isBlank) StreamProperties(org.apache.bookkeeper.stream.proto.StreamProperties) CompareResult(org.apache.bookkeeper.api.kv.op.CompareResult) ProtoUtils.validateNamespaceName(org.apache.bookkeeper.stream.protocol.util.ProtoUtils.validateNamespaceName) GetNamespaceRequest(org.apache.bookkeeper.stream.proto.storage.GetNamespaceRequest) GetNamespaceResponse(org.apache.bookkeeper.stream.proto.storage.GetNamespaceResponse) StreamConfiguration(org.apache.bookkeeper.stream.proto.StreamConfiguration) StreamProperties(org.apache.bookkeeper.stream.proto.StreamProperties)

Example 3 with TxnOp

use of org.apache.bookkeeper.api.kv.op.TxnOp in project bookkeeper by apache.

the class MVCCStoreImpl method processTxn.

synchronized TxnResult<K, V> processTxn(long revision, TxnOp<K, V> op) {
    checkStoreOpen();
    // 1. process the compares
    boolean success = processCompares(op);
    // 2. prepare the response list
    List<Op<K, V>> operations;
    List<Result<K, V>> results;
    if (success) {
        operations = op.successOps();
    } else {
        operations = op.failureOps();
    }
    if (operations == null) {
        operations = Collections.emptyList();
    }
    results = Lists.newArrayListWithExpectedSize(operations.size());
    // 3. process the operations
    try (WriteBatch batch = new WriteBatch()) {
        for (Op<K, V> o : operations) {
            results.add(executeOp(revision, batch, o));
        }
        executeBatch(batch);
        // 4. repare the result
        TxnResultImpl<K, V> txnResult = resultFactory.newTxnResult(revision);
        txnResult.isSuccess(success);
        txnResult.results(results);
        txnResult.code(Code.OK);
        return txnResult;
    } catch (StateStoreRuntimeException e) {
        results.forEach(Result::close);
        throw e;
    }
}
Also used : TxnOp(org.apache.bookkeeper.api.kv.op.TxnOp) CompareOp(org.apache.bookkeeper.api.kv.op.CompareOp) PutOp(org.apache.bookkeeper.api.kv.op.PutOp) Op(org.apache.bookkeeper.api.kv.op.Op) IncrementOp(org.apache.bookkeeper.api.kv.op.IncrementOp) RangeOp(org.apache.bookkeeper.api.kv.op.RangeOp) DeleteOp(org.apache.bookkeeper.api.kv.op.DeleteOp) KV(org.apache.bookkeeper.common.kv.KV) WriteBatch(org.rocksdb.WriteBatch) StateStoreRuntimeException(org.apache.bookkeeper.statelib.api.exceptions.StateStoreRuntimeException) PutResult(org.apache.bookkeeper.api.kv.result.PutResult) Result(org.apache.bookkeeper.api.kv.result.Result) IncrementResult(org.apache.bookkeeper.api.kv.result.IncrementResult) RangeResult(org.apache.bookkeeper.api.kv.result.RangeResult) TxnResult(org.apache.bookkeeper.api.kv.result.TxnResult) DeleteResult(org.apache.bookkeeper.api.kv.result.DeleteResult) CompareResult(org.apache.bookkeeper.api.kv.op.CompareResult)

Example 4 with TxnOp

use of org.apache.bookkeeper.api.kv.op.TxnOp in project bookkeeper by apache.

the class MetaRangeImpl method unsafeCreate.

private void unsafeCreate(CompletableFuture<Boolean> createFuture, StreamProperties streamProps) {
    // 1. verify the state
    checkLifecycleState(LifecycleState.UNINIT);
    this.lifecycleState = LifecycleState.CREATING;
    // 2. store the props/configuration
    synchronized (this) {
        this.streamProps = streamProps;
    }
    this.streamId = streamProps.getStreamId();
    this.cTime = this.mTime = System.currentTimeMillis();
    // 3. create the ranges
    List<RangeProperties> propertiesList = split(streamId, streamProps.getStreamConf().getInitialNumRanges(), nextRangeId, placementPolicy);
    List<Op<byte[], byte[]>> successOps = Lists.newArrayListWithExpectedSize(propertiesList.size() + 1);
    for (RangeProperties props : propertiesList) {
        RangeMetadata meta = RangeMetadata.newBuilder().setProps(props).setCreateTime(cTime).setFenceTime(Long.MAX_VALUE).setState(RangeState.RANGE_ACTIVE).addAllParents(Lists.newArrayList()).build();
        ranges.put(props.getRangeId(), meta);
        currentRanges.add(props.getRangeId());
        successOps.add(store.newPut(getStreamRangeKey(streamId, props.getRangeId()), meta.toByteArray()));
    }
    nextRangeId += propertiesList.size();
    // serialize the stream metadata
    byte[] streamMetadataKey = getStreamMetadataKey(streamId);
    successOps.add(store.newPut(streamMetadataKey, toStreamMetadata(LifecycleState.CREATED).toByteArray()));
    TxnOp<byte[], byte[]> txn = store.newTxn().If(store.newCompareValue(CompareResult.EQUAL, streamMetadataKey, null)).Then(successOps.toArray(new Op[successOps.size()])).build();
    if (log.isTraceEnabled()) {
        log.trace("Execute create stream metadata range txn {}", streamProps);
    }
    store.txn(txn).thenApplyAsync(txnResult -> {
        try {
            if (log.isTraceEnabled()) {
                log.trace("Create stream metadata range txn result = {}", txnResult.isSuccess());
            }
            if (txnResult.isSuccess()) {
                List<Result<byte[], byte[]>> results = txnResult.results();
                MetaRangeImpl.this.revision = results.get(results.size() - 1).revision();
                // mark the state to CREATED
                this.lifecycleState = LifecycleState.CREATED;
                createFuture.complete(true);
            } else {
                createFuture.complete(false);
            }
            return null;
        } finally {
            txnResult.close();
        }
    }, executor).exceptionally(cause -> {
        createFuture.completeExceptionally(cause);
        return null;
    });
}
Also used : RangeMetadata(org.apache.bookkeeper.stream.proto.RangeMetadata) RangeState(org.apache.bookkeeper.stream.proto.RangeState) CompletableFuture(java.util.concurrent.CompletableFuture) Supplier(java.util.function.Supplier) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) ProtoUtils.split(org.apache.bookkeeper.stream.protocol.util.ProtoUtils.split) MVCCAsyncStore(org.apache.bookkeeper.statelib.api.mvcc.MVCCAsyncStore) StreamMetadata(org.apache.bookkeeper.stream.proto.StreamMetadata) Lists(com.google.common.collect.Lists) TxnOp(org.apache.bookkeeper.api.kv.op.TxnOp) MetaRange(org.apache.bookkeeper.stream.storage.api.metadata.stream.MetaRange) ImmutableList(com.google.common.collect.ImmutableList) ProtoUtils.isStreamCreated(org.apache.bookkeeper.stream.protocol.util.ProtoUtils.isStreamCreated) RangeProperties(org.apache.bookkeeper.stream.proto.RangeProperties) LifecycleState(org.apache.bookkeeper.stream.proto.StreamMetadata.LifecycleState) KeyValue(org.apache.bookkeeper.api.kv.result.KeyValue) ExecutorService(java.util.concurrent.ExecutorService) InvalidProtocolBufferException(com.google.protobuf.InvalidProtocolBufferException) StorageContainerPlacementPolicy(org.apache.bookkeeper.stream.protocol.util.StorageContainerPlacementPolicy) MIN_DATA_RANGE_ID(org.apache.bookkeeper.stream.protocol.ProtocolConstants.MIN_DATA_RANGE_ID) DataRangeNotFoundException(org.apache.bookkeeper.stream.storage.exceptions.DataRangeNotFoundException) ServingState(org.apache.bookkeeper.stream.proto.StreamMetadata.ServingState) Result(org.apache.bookkeeper.api.kv.result.Result) StreamConfiguration(org.apache.bookkeeper.stream.proto.StreamConfiguration) Bytes(org.apache.bookkeeper.common.util.Bytes) NavigableMap(java.util.NavigableMap) GuardedBy(javax.annotation.concurrent.GuardedBy) FutureUtils(org.apache.bookkeeper.common.concurrent.FutureUtils) Maps(com.google.common.collect.Maps) Preconditions.checkState(com.google.common.base.Preconditions.checkState) Consumer(java.util.function.Consumer) List(java.util.List) Slf4j(lombok.extern.slf4j.Slf4j) StreamProperties(org.apache.bookkeeper.stream.proto.StreamProperties) VisibleForTesting(com.google.common.annotations.VisibleForTesting) CompareResult(org.apache.bookkeeper.api.kv.op.CompareResult) Op(org.apache.bookkeeper.api.kv.op.Op) RangeProperties(org.apache.bookkeeper.stream.proto.RangeProperties) TxnOp(org.apache.bookkeeper.api.kv.op.TxnOp) Op(org.apache.bookkeeper.api.kv.op.Op) ImmutableList(com.google.common.collect.ImmutableList) List(java.util.List) RangeMetadata(org.apache.bookkeeper.stream.proto.RangeMetadata)

Aggregations

CompareResult (org.apache.bookkeeper.api.kv.op.CompareResult)4 TxnOp (org.apache.bookkeeper.api.kv.op.TxnOp)4 InvalidProtocolBufferException (com.google.protobuf.InvalidProtocolBufferException)3 CompletableFuture (java.util.concurrent.CompletableFuture)3 Slf4j (lombok.extern.slf4j.Slf4j)3 RangeOp (org.apache.bookkeeper.api.kv.op.RangeOp)3 KeyValue (org.apache.bookkeeper.api.kv.result.KeyValue)3 FutureUtils (org.apache.bookkeeper.common.concurrent.FutureUtils)3 Bytes (org.apache.bookkeeper.common.util.Bytes)3 MVCCAsyncStore (org.apache.bookkeeper.statelib.api.mvcc.MVCCAsyncStore)3 StreamConfiguration (org.apache.bookkeeper.stream.proto.StreamConfiguration)3 StreamProperties (org.apache.bookkeeper.stream.proto.StreamProperties)3 StorageContainerPlacementPolicy (org.apache.bookkeeper.stream.protocol.util.StorageContainerPlacementPolicy)3 UTF_8 (com.google.common.base.Charsets.UTF_8)2 URI (java.net.URI)2 ScheduledExecutorService (java.util.concurrent.ScheduledExecutorService)2 Options (org.apache.bookkeeper.api.kv.options.Options)2 NamespaceMetadata (org.apache.bookkeeper.stream.proto.NamespaceMetadata)2 NamespaceProperties (org.apache.bookkeeper.stream.proto.NamespaceProperties)2 StreamName (org.apache.bookkeeper.stream.proto.StreamName)2