Examples with StreamSegmentNotExistsException io.pravega.segmentstore.contracts.StreamSegmentNotExistsException used on opensource projects

Example 41 with StreamSegmentNotExistsException

use of io.pravega.segmentstore.contracts.StreamSegmentNotExistsException in project pravega by pravega.

the class RollingStorage method delete.

@Override
public void delete(SegmentHandle handle) throws StreamSegmentException {
    val h = asReadableHandle(handle);
    long traceId = LoggerHelpers.traceEnter(log, "delete", handle);
    SegmentHandle headerHandle = h.getHeaderHandle();
    if (headerHandle == null) {
        // Directly delete the only SegmentChunk, and bubble up any exceptions if it doesn't exist.
        val subHandle = this.baseStorage.openWrite(h.lastChunk().getName());
        try {
            this.baseStorage.delete(subHandle);
            h.lastChunk().markInexistent();
            h.markDeleted();
        } catch (StreamSegmentNotExistsException ex) {
            h.lastChunk().markInexistent();
            h.markDeleted();
            throw ex;
        }
    } else {
        // them, after which we delete all SegmentChunks and finally the header file.
        if (!h.isSealed()) {
            val writeHandle = h.isReadOnly() ? (RollingSegmentHandle) openWrite(handle.getSegmentName()) : h;
            seal(writeHandle);
        }
        deleteChunks(h, s -> true);
        try {
            this.baseStorage.delete(headerHandle);
            h.markDeleted();
        } catch (StreamSegmentNotExistsException ex) {
            h.markDeleted();
            throw ex;
        }
    }
    LoggerHelpers.traceLeave(log, "delete", traceId, handle);
}

Example 42 with StreamSegmentNotExistsException

use of io.pravega.segmentstore.contracts.StreamSegmentNotExistsException in project pravega by pravega.

the class SegmentAggregator method initialize.

// endregion
// region Operations
/**
 * Initializes the SegmentAggregator by pulling information from the given Storage.
 *
 * @param timeout  Timeout for the operation.
 * @return A CompletableFuture that, when completed, will indicate that the operation finished successfully. If any
 * errors occurred during the operation, the Future will be completed with the appropriate exception.
 */
CompletableFuture<Void> initialize(Duration timeout) {
    Exceptions.checkNotClosed(isClosed(), this);
    Preconditions.checkState(this.state.get() == AggregatorState.NotInitialized, "SegmentAggregator has already been initialized.");
    assert this.handle.get() == null : "non-null handle but state == " + this.state.get();
    long traceId = LoggerHelpers.traceEnterWithContext(log, this.traceObjectId, "initialize");
    return openWrite(this.metadata.getName(), this.handle, timeout).thenAcceptAsync(segmentInfo -> {
        // Check & Update StorageLength in metadata.
        if (this.metadata.getStorageLength() != segmentInfo.getLength()) {
            if (this.metadata.getStorageLength() >= 0) {
                // Only log warning if the StorageLength has actually been initialized, but is different.
                log.warn("{}: SegmentMetadata has a StorageLength ({}) that is different than the actual one ({}) - updating metadata.", this.traceObjectId, this.metadata.getStorageLength(), segmentInfo.getLength());
            }
            // It is very important to keep this value up-to-date and correct.
            this.metadata.setStorageLength(segmentInfo.getLength());
        }
        // Check if the Storage segment is sealed, but it's not in metadata (this is 100% indicative of some data corruption happening).
        if (segmentInfo.isSealed()) {
            if (!this.metadata.isSealed()) {
                throw new CompletionException(new DataCorruptionException(String.format("Segment '%s' is sealed in Storage but not in the metadata.", this.metadata.getName())));
            }
            if (!this.metadata.isSealedInStorage()) {
                this.metadata.markSealedInStorage();
                log.warn("{}: Segment is sealed in Storage but metadata does not reflect that - updating metadata.", this.traceObjectId);
            }
        }
        log.info("{}: Initialized. StorageLength = {}, Sealed = {}.", this.traceObjectId, segmentInfo.getLength(), segmentInfo.isSealed());
        LoggerHelpers.traceLeave(log, this.traceObjectId, "initialize", traceId);
        setState(AggregatorState.Writing);
    }, this.executor).exceptionally(ex -> {
        ex = Exceptions.unwrap(ex);
        if (ex instanceof StreamSegmentNotExistsException) {
            // Segment does not exist anymore. This is a real possibility during recovery, in the following cases:
            // * We already processed a Segment Deletion but did not have a chance to checkpoint metadata
            // * We processed a TransactionMergeOperation but did not have a chance to ack/truncate the DataSource
            // Update metadata, just in case it is not already updated.
            this.metadata.markDeleted();
            log.warn("{}: Segment does not exist in Storage. Ignoring all further operations on it.", this.traceObjectId, ex);
            setState(AggregatorState.Writing);
            LoggerHelpers.traceLeave(log, this.traceObjectId, "initialize", traceId);
        } else {
            // Other kind of error - re-throw.
            throw new CompletionException(ex);
        }
        return null;
    });
}

Example 43 with StreamSegmentNotExistsException

use of io.pravega.segmentstore.contracts.StreamSegmentNotExistsException in project pravega by pravega.

the class SegmentAggregator method mergeWith.

/**
 * Merges the Transaction StreamSegment with given metadata into this one at the current offset.
 *
 * @param transactionMetadata The metadata of the Transaction StreamSegment to merge.
 * @param timer               Timer for the operation.
 * @return A CompletableFuture that, when completed, will contain the number of bytes that were merged into this
 * StreamSegment. If failed, the Future will contain the exception that caused it.
 */
private CompletableFuture<FlushResult> mergeWith(UpdateableSegmentMetadata transactionMetadata, MergeTransactionOperation mergeOp, TimeoutTimer timer) {
    if (transactionMetadata.isDeleted()) {
        return Futures.failedFuture(new DataCorruptionException(String.format("Attempted to merge with deleted Transaction segment '%s'.", transactionMetadata.getName())));
    }
    long traceId = LoggerHelpers.traceEnterWithContext(log, this.traceObjectId, "mergeWith", transactionMetadata.getId(), transactionMetadata.getName(), transactionMetadata.isSealedInStorage());
    FlushResult result = new FlushResult();
    if (!transactionMetadata.isSealedInStorage() || transactionMetadata.getLength() > transactionMetadata.getStorageLength()) {
        // Nothing to do. Given Transaction is not eligible for merger yet.
        LoggerHelpers.traceLeave(log, this.traceObjectId, "mergeWith", traceId, result);
        return CompletableFuture.completedFuture(result);
    }
    AtomicLong mergedLength = new AtomicLong();
    return this.storage.getStreamSegmentInfo(transactionMetadata.getName(), timer.getRemaining()).thenAcceptAsync(transProperties -> {
        // Check that the Storage agrees with our metadata (if not, we have a problem ...)
        if (transProperties.getLength() != transactionMetadata.getStorageLength()) {
            throw new CompletionException(new DataCorruptionException(String.format("Transaction Segment '%s' cannot be merged into parent '%s' because its metadata disagrees with the Storage. Metadata.StorageLength=%d, Storage.StorageLength=%d", transactionMetadata.getName(), this.metadata.getName(), transactionMetadata.getStorageLength(), transProperties.getLength())));
        }
        if (transProperties.getLength() != mergeOp.getLength()) {
            throw new CompletionException(new DataCorruptionException(String.format("Transaction Segment '%s' cannot be merged into parent '%s' because the declared length in the operation disagrees with the Storage. Operation.Length=%d, Storage.StorageLength=%d", transactionMetadata.getName(), this.metadata.getName(), mergeOp.getLength(), transProperties.getLength())));
        }
        mergedLength.set(transProperties.getLength());
    }, this.executor).thenComposeAsync(v1 -> storage.concat(this.handle.get(), mergeOp.getStreamSegmentOffset(), transactionMetadata.getName(), timer.getRemaining()), this.executor).thenComposeAsync(v2 -> storage.getStreamSegmentInfo(this.metadata.getName(), timer.getRemaining()), this.executor).thenApplyAsync(segmentProperties -> {
        // We have processed a MergeTransactionOperation, pop the first operation off and decrement the counter.
        StorageOperation processedOperation = this.operations.removeFirst();
        assert processedOperation != null && processedOperation instanceof MergeTransactionOperation : "First outstanding operation was not a MergeTransactionOperation";
        assert ((MergeTransactionOperation) processedOperation).getTransactionSegmentId() == transactionMetadata.getId() : "First outstanding operation was a MergeTransactionOperation for the wrong Transaction id.";
        int newCount = this.mergeTransactionCount.decrementAndGet();
        assert newCount >= 0 : "Negative value for mergeTransactionCount";
        // Post-merger validation. Verify we are still in agreement with the storage.
        long expectedNewLength = this.metadata.getStorageLength() + mergedLength.get();
        if (segmentProperties.getLength() != expectedNewLength) {
            throw new CompletionException(new DataCorruptionException(String.format("Transaction Segment '%s' was merged into parent '%s' but the parent segment has an unexpected StorageLength after the merger. Previous=%d, MergeLength=%d, Expected=%d, Actual=%d", transactionMetadata.getName(), this.metadata.getName(), segmentProperties.getLength(), mergedLength.get(), expectedNewLength, segmentProperties.getLength())));
        }
        updateMetadata(segmentProperties);
        updateMetadataForTransactionPostMerger(transactionMetadata);
        this.lastFlush.set(this.timer.getElapsed());
        result.withMergedBytes(mergedLength.get());
        LoggerHelpers.traceLeave(log, this.traceObjectId, "mergeWith", traceId, result);
        return result;
    }, this.executor).exceptionally(ex -> {
        Throwable realEx = Exceptions.unwrap(ex);
        if (realEx instanceof BadOffsetException || realEx instanceof StreamSegmentNotExistsException) {
            // We either attempted to write at an offset that already contained other data or the Transaction
            // Segment no longer exists. This can happen for a number of reasons, but we do not have enough
            // information here to determine why. We need to enter reconciliation mode, and hope for the best.
            setState(AggregatorState.ReconciliationNeeded);
        }
        // Rethrow all exceptions.
        throw new CompletionException(ex);
    });
}

Example 44 with StreamSegmentNotExistsException

use of io.pravega.segmentstore.contracts.StreamSegmentNotExistsException in project pravega by pravega.

the class ReadOnlySegmentContainerTests method testReadInexistentSegment.

/**
 * Tests the read() method when the segment does not exist.
 */
@Test
public void testReadInexistentSegment() {
    @Cleanup val context = new TestContext();
    context.container.startAsync().awaitRunning();
    AssertExtensions.assertThrows("Unexpected exception when the segment does not exist.", () -> context.container.read(SEGMENT_NAME, 0, 1, TIMEOUT), ex -> ex instanceof StreamSegmentNotExistsException);
}

Example 45 with StreamSegmentNotExistsException

use of io.pravega.segmentstore.contracts.StreamSegmentNotExistsException in project pravega by pravega.

the class ReadOnlySegmentContainerTests method testGetStreamSegmentInfo.

/**
 * Tests the getStreamSegmentInfo() method.
 */
@Test
public void testGetStreamSegmentInfo() {
    @Cleanup val context = new TestContext();
    context.container.startAsync().awaitRunning();
    // Non-existent segment.
    AssertExtensions.assertThrows("Unexpected exception when the segment does not exist.", () -> context.container.getStreamSegmentInfo(SEGMENT_NAME, false, TIMEOUT), ex -> ex instanceof StreamSegmentNotExistsException);
    // Create a segment, add some data, set some attributes, "truncate" it and then seal it.
    val storageInfo = context.storage.create(SEGMENT_NAME, TIMEOUT).thenCompose(si -> context.storage.openWrite(si.getName())).thenCompose(handle -> context.storage.write(handle, 0, new ByteArrayInputStream(new byte[10]), 10, TIMEOUT)).thenCompose(v -> context.storage.getStreamSegmentInfo(SEGMENT_NAME, TIMEOUT)).join();
    val expectedInfo = StreamSegmentInformation.from(storageInfo).startOffset(storageInfo.getLength() / 2).attributes(Collections.singletonMap(UUID.randomUUID(), 100L)).build();
    context.stateStore.put(SEGMENT_NAME, new SegmentState(1, expectedInfo), TIMEOUT).join();
    // Fetch the SegmentInfo from the ReadOnlyContainer and verify it is as expected.
    val actual = context.container.getStreamSegmentInfo(SEGMENT_NAME, false, TIMEOUT).join();
    Assert.assertEquals("Unexpected Name.", expectedInfo.getName(), actual.getName());
    Assert.assertEquals("Unexpected Length.", expectedInfo.getLength(), actual.getLength());
    Assert.assertEquals("Unexpected Sealed status.", expectedInfo.isSealed(), actual.isSealed());
    Assert.assertEquals("Unexpected Start Offset.", expectedInfo.getStartOffset(), actual.getStartOffset());
    AssertExtensions.assertMapEquals("Unexpected Attributes.", expectedInfo.getAttributes(), actual.getAttributes());
}