Search in sources :

Example 46 with SegmentRollingPolicy

use of io.pravega.segmentstore.storage.SegmentRollingPolicy in project pravega by pravega.

the class SystemJournalTests method testSimpleBootstrapWithTwoTruncates.

/**
 * Tests a scenario when there are two fail overs.
 * The test adds a few chunks to the system segments and then fails over.
 * It also truncates system segments.
 * The new instance should read the journal log file and recreate the layout of system segments.
 *
 * @throws Exception Throws exception in case of any error.
 */
@Test
public void testSimpleBootstrapWithTwoTruncates() throws Exception {
    @Cleanup CleanupHelper cleanupHelper = new CleanupHelper();
    @Cleanup ChunkStorage chunkStorage = getChunkStorage();
    @Cleanup ChunkMetadataStore metadataStoreBeforeCrash = getMetadataStore();
    @Cleanup ChunkMetadataStore metadataStoreAfterCrash = getMetadataStore();
    int containerId = 42;
    String systemSegmentName = SystemJournal.getChunkStorageSystemSegments(containerId)[0];
    long epoch = 1;
    val policy = new SegmentRollingPolicy(8);
    val config = getDefaultConfigBuilder(policy).build();
    long offset = 0;
    val data = new InMemorySnapshotInfoStore();
    val snapshotInfoStore = new SnapshotInfoStore(containerId, snapshotId -> data.setSnapshotId(containerId, snapshotId), () -> data.getSnapshotId(containerId));
    // Epoch 1
    @Cleanup ChunkedSegmentStorage segmentStorage1 = new ChunkedSegmentStorage(containerId, chunkStorage, metadataStoreBeforeCrash, executorService(), config);
    segmentStorage1.initialize(epoch);
    segmentStorage1.getGarbageCollector().initialize(new InMemoryTaskQueueManager()).join();
    // Bootstrap
    segmentStorage1.bootstrap(snapshotInfoStore, null).join();
    deleteGarbage(segmentStorage1);
    checkSystemSegmentsLayout(segmentStorage1);
    // Simulate some writes to system segment, this should cause some new chunks being added.
    val h = segmentStorage1.openWrite(systemSegmentName).join();
    segmentStorage1.write(h, offset, new ByteArrayInputStream("JUNKJUNKJUNK".getBytes()), 12, null).join();
    offset += 12;
    val b1 = "Hello".getBytes();
    segmentStorage1.write(h, offset, new ByteArrayInputStream(b1), b1.length, null).join();
    offset += b1.length;
    segmentStorage1.truncate(h, 6, null).join();
    // Epoch 2
    epoch++;
    @Cleanup ChunkedSegmentStorage segmentStorage2 = new ChunkedSegmentStorage(containerId, chunkStorage, metadataStoreAfterCrash, executorService(), config);
    segmentStorage2.initialize(epoch);
    segmentStorage2.getGarbageCollector().initialize(new InMemoryTaskQueueManager()).join();
    segmentStorage2.bootstrap(snapshotInfoStore, null).join();
    deleteGarbage(segmentStorage2);
    checkSystemSegmentsLayout(segmentStorage2);
    val h2 = segmentStorage2.openWrite(systemSegmentName).join();
    // Write Junk data to both system segment and journal file.
    segmentStorage1.write(h, offset, new ByteArrayInputStream("junk".getBytes()), 4, null).join();
    val b2 = " World".getBytes();
    segmentStorage2.write(h2, offset, new ByteArrayInputStream(b2), b2.length, null).join();
    offset += b2.length;
    segmentStorage2.truncate(h, 12, null).join();
    val info = segmentStorage2.getStreamSegmentInfo(systemSegmentName, null).join();
    Assert.assertEquals(b1.length + b2.length + 12, info.getLength());
    Assert.assertEquals(12, info.getStartOffset());
    byte[] out = new byte[b1.length + b2.length];
    val hr = segmentStorage2.openRead(systemSegmentName).join();
    segmentStorage2.read(hr, 12, out, 0, b1.length + b2.length, null).join();
    Assert.assertEquals("Hello World", new String(out));
}
Also used : lombok.val(lombok.val) SegmentRollingPolicy(io.pravega.segmentstore.storage.SegmentRollingPolicy) InMemoryChunkStorage(io.pravega.segmentstore.storage.mocks.InMemoryChunkStorage) Cleanup(lombok.Cleanup) InMemoryTaskQueueManager(io.pravega.segmentstore.storage.mocks.InMemoryTaskQueueManager) ByteArrayInputStream(java.io.ByteArrayInputStream) InMemorySnapshotInfoStore(io.pravega.segmentstore.storage.mocks.InMemorySnapshotInfoStore) InMemorySnapshotInfoStore(io.pravega.segmentstore.storage.mocks.InMemorySnapshotInfoStore) ChunkMetadataStore(io.pravega.segmentstore.storage.metadata.ChunkMetadataStore) Test(org.junit.Test)

Example 47 with SegmentRollingPolicy

use of io.pravega.segmentstore.storage.SegmentRollingPolicy in project pravega by pravega.

the class SystemJournalTests method testSimpleBootstrapWithTwoFailovers.

/**
 * Tests a scenario when there are two fail overs.
 * The test adds a few chunks to the system segments and then fails over.
 * After fail over the zombie instance continues to write junk data to both system segment and journal file.
 * The new instance should read the journal log file and recreate the layout of system segments.
 *
 * @throws Exception Throws exception in case of any error.
 */
@Test
public void testSimpleBootstrapWithTwoFailovers() throws Exception {
    @Cleanup ChunkStorage chunkStorage = getChunkStorage();
    @Cleanup ChunkMetadataStore metadataStoreBeforeCrash = getMetadataStore();
    @Cleanup ChunkMetadataStore metadataStoreAfterCrash = getMetadataStore();
    int containerId = 42;
    String systemSegmentName = SystemJournal.getChunkStorageSystemSegments(containerId)[0];
    long epoch = 1;
    val policy = new SegmentRollingPolicy(8);
    val config = getDefaultConfigBuilder(policy).build();
    val data = new InMemorySnapshotInfoStore();
    val snapshotInfoStore = new SnapshotInfoStore(containerId, snapshotId -> data.setSnapshotId(containerId, snapshotId), () -> data.getSnapshotId(containerId));
    long offset = 0;
    // Epoch 1
    @Cleanup ChunkedSegmentStorage segmentStorage1 = new ChunkedSegmentStorage(containerId, chunkStorage, metadataStoreBeforeCrash, executorService(), config);
    segmentStorage1.initialize(epoch);
    segmentStorage1.getGarbageCollector().initialize(new InMemoryTaskQueueManager()).join();
    // Bootstrap
    segmentStorage1.bootstrap(snapshotInfoStore, null).join();
    deleteGarbage(segmentStorage1);
    checkSystemSegmentsLayout(segmentStorage1);
    // Simulate some writes to system segment, this should cause some new chunks being added.
    val h = segmentStorage1.openWrite(systemSegmentName).join();
    val b1 = "Hello".getBytes();
    segmentStorage1.write(h, offset, new ByteArrayInputStream(b1), b1.length, null).join();
    offset += b1.length;
    checkSystemSegmentsLayout(segmentStorage1);
    // Epoch 2
    epoch++;
    @Cleanup ChunkedSegmentStorage segmentStorage2 = new ChunkedSegmentStorage(containerId, chunkStorage, metadataStoreAfterCrash, executorService(), config);
    segmentStorage2.initialize(epoch);
    segmentStorage2.getGarbageCollector().initialize(new InMemoryTaskQueueManager()).join();
    // Bootstrap
    segmentStorage2.bootstrap(snapshotInfoStore, null).join();
    deleteGarbage(segmentStorage2);
    checkSystemSegmentsLayout(segmentStorage2);
    val h2 = segmentStorage2.openWrite(systemSegmentName).join();
    // Write Junk Data to from first instance.
    segmentStorage1.write(h, offset, new ByteArrayInputStream("junk".getBytes()), 4, null).join();
    val b2 = " World".getBytes();
    segmentStorage2.write(h2, offset, new ByteArrayInputStream(b2), b2.length, null).join();
    offset += b2.length;
    checkSystemSegmentsLayout(segmentStorage2);
    val info = segmentStorage2.getStreamSegmentInfo(systemSegmentName, null).join();
    Assert.assertEquals(b1.length + b2.length, info.getLength());
    byte[] out = new byte[b1.length + b2.length];
    val hr = segmentStorage2.openRead(systemSegmentName).join();
    segmentStorage2.read(hr, 0, out, 0, b1.length + b2.length, null).join();
    Assert.assertEquals("Hello World", new String(out));
}
Also used : lombok.val(lombok.val) SegmentRollingPolicy(io.pravega.segmentstore.storage.SegmentRollingPolicy) InMemoryChunkStorage(io.pravega.segmentstore.storage.mocks.InMemoryChunkStorage) Cleanup(lombok.Cleanup) InMemoryTaskQueueManager(io.pravega.segmentstore.storage.mocks.InMemoryTaskQueueManager) ByteArrayInputStream(java.io.ByteArrayInputStream) InMemorySnapshotInfoStore(io.pravega.segmentstore.storage.mocks.InMemorySnapshotInfoStore) InMemorySnapshotInfoStore(io.pravega.segmentstore.storage.mocks.InMemorySnapshotInfoStore) ChunkMetadataStore(io.pravega.segmentstore.storage.metadata.ChunkMetadataStore) Test(org.junit.Test)

Example 48 with SegmentRollingPolicy

use of io.pravega.segmentstore.storage.SegmentRollingPolicy in project pravega by pravega.

the class SystemJournalTests method testSimpleBootstrapWithMissingSnapshot.

@Test
public void testSimpleBootstrapWithMissingSnapshot() throws Exception {
    val containerId = 42;
    @Cleanup ChunkStorage chunkStorage = getChunkStorage();
    val policy = new SegmentRollingPolicy(100);
    val config = getDefaultConfigBuilder(policy).selfCheckEnabled(true).build();
    try {
        testSimpleBootstrapWithMultipleFailovers(containerId, chunkStorage, config, epoch -> {
            val snapShotFile = NameUtils.getSystemJournalSnapshotFileName(containerId, epoch, 1);
            chunkStorage.delete(ChunkHandle.writeHandle(snapShotFile)).join();
        });
    } catch (Exception e) {
        val ex = Exceptions.unwrap(e);
        Assert.assertTrue(Exceptions.unwrap(e) instanceof IllegalStateException && ex.getMessage().contains("Chunk pointed by SnapshotInfo must exist"));
    }
}
Also used : lombok.val(lombok.val) SegmentRollingPolicy(io.pravega.segmentstore.storage.SegmentRollingPolicy) InMemoryChunkStorage(io.pravega.segmentstore.storage.mocks.InMemoryChunkStorage) Cleanup(lombok.Cleanup) Test(org.junit.Test)

Example 49 with SegmentRollingPolicy

use of io.pravega.segmentstore.storage.SegmentRollingPolicy in project pravega by pravega.

the class RollingStorageTestBase method testWriteOnRollOverBoundary.

@Test
public void testWriteOnRollOverBoundary() throws Exception {
    final String segmentName = "Segment";
    // Really small rolling length.
    final int maxLength = 3;
    val seq1 = "01234";
    val seq2 = "56789";
    val totalWriteLength = seq1.length() + seq2.length();
    @Cleanup val s = createStorage();
    s.initialize(1);
    val writeHandle = s.create(segmentName, new SegmentRollingPolicy(maxLength), TIMEOUT).thenCompose(v -> s.openWrite(segmentName)).join();
    val byteInputStream1 = new ByteArrayInputStream(seq1.getBytes());
    val byteInputStream2 = new ByteArrayInputStream(seq2.getBytes());
    val sequenceInputStream = new SequenceInputStream(byteInputStream1, byteInputStream2);
    // This write should cause 3 rollovers.
    s.write(writeHandle, 0, sequenceInputStream, totalWriteLength, TIMEOUT).join();
    // Check rollover actually happened as expected.
    if (useOldLayout) {
        RollingSegmentHandle checkHandle = (RollingSegmentHandle) s.openWrite(segmentName).join();
        val chunks = checkHandle.chunks();
        int numberOfRollovers = totalWriteLength / maxLength;
        Assert.assertEquals(numberOfRollovers + 1, chunks.size());
        for (int i = 0; i < numberOfRollovers; i++) {
            Assert.assertEquals(maxLength * i, chunks.get(i).getStartOffset());
            Assert.assertEquals(maxLength, chunks.get(i).getLength());
        }
        // Last chunk has index == numberOfRollovers, as list is 0 based.
        Assert.assertEquals(numberOfRollovers * maxLength, chunks.get(numberOfRollovers).getStartOffset());
        Assert.assertEquals(1, chunks.get(numberOfRollovers).getLength());
        // Now validate the contents written.
        val readHandle = s.openRead(segmentName).join();
        byte[] output = new byte[totalWriteLength];
        s.read(readHandle, 0, output, 0, totalWriteLength, TIMEOUT).join();
        Assert.assertEquals(seq1 + seq2, new String(output));
    }
}
Also used : lombok.val(lombok.val) Storage(io.pravega.segmentstore.storage.Storage) ByteArrayOutputStream(java.io.ByteArrayOutputStream) SegmentRollingPolicy(io.pravega.segmentstore.storage.SegmentRollingPolicy) SequenceInputStream(java.io.SequenceInputStream) lombok.val(lombok.val) AsyncStorageWrapper(io.pravega.segmentstore.storage.AsyncStorageWrapper) Cleanup(lombok.Cleanup) Random(java.util.Random) Test(org.junit.Test) ByteArrayInputStream(java.io.ByteArrayInputStream) StorageTestBase(io.pravega.segmentstore.storage.StorageTestBase) SyncStorage(io.pravega.segmentstore.storage.SyncStorage) Assert(org.junit.Assert) SegmentRollingPolicy(io.pravega.segmentstore.storage.SegmentRollingPolicy) SequenceInputStream(java.io.SequenceInputStream) ByteArrayInputStream(java.io.ByteArrayInputStream) Cleanup(lombok.Cleanup) Test(org.junit.Test)

Example 50 with SegmentRollingPolicy

use of io.pravega.segmentstore.storage.SegmentRollingPolicy in project pravega by pravega.

the class ChunkedSegmentStorage method create.

@Override
public CompletableFuture<SegmentHandle> create(String streamSegmentName, SegmentRollingPolicy rollingPolicy, Duration timeout) {
    checkInitialized();
    return executeSerialized(() -> {
        val traceId = LoggerHelpers.traceEnter(log, "create", streamSegmentName, rollingPolicy);
        val timer = new Timer();
        log.debug("{} create - started segment={}, rollingPolicy={}.", logPrefix, streamSegmentName, rollingPolicy);
        return tryWith(metadataStore.beginTransaction(false, streamSegmentName), txn -> {
            // Retrieve metadata and make sure it does not exist.
            return txn.get(streamSegmentName).thenComposeAsync(storageMetadata -> {
                val oldSegmentMetadata = (SegmentMetadata) storageMetadata;
                if (null != oldSegmentMetadata) {
                    throw new CompletionException(new StreamSegmentExistsException(streamSegmentName));
                }
                // Create a new record.
                val newSegmentMetadata = SegmentMetadata.builder().name(streamSegmentName).maxRollinglength(rollingPolicy.getMaxLength() == 0 ? SegmentRollingPolicy.NO_ROLLING.getMaxLength() : rollingPolicy.getMaxLength()).ownerEpoch(this.epoch).build();
                newSegmentMetadata.setActive(true);
                txn.create(newSegmentMetadata);
                // commit.
                return txn.commit().thenApplyAsync(v -> {
                    val retValue = SegmentStorageHandle.writeHandle(streamSegmentName);
                    Duration elapsed = timer.getElapsed();
                    SLTS_CREATE_LATENCY.reportSuccessEvent(elapsed);
                    SLTS_CREATE_COUNT.inc();
                    log.debug("{} create - finished segment={}, rollingPolicy={}, latency={}.", logPrefix, streamSegmentName, rollingPolicy, elapsed.toMillis());
                    LoggerHelpers.traceLeave(log, "create", traceId, retValue);
                    return retValue;
                }, executor);
            }, executor);
        }, executor).handleAsync((v, e) -> {
            if (null != e) {
                log.debug("{} create - exception segment={}, rollingPolicy={}, latency={}.", logPrefix, streamSegmentName, rollingPolicy, timer.getElapsedMillis(), e);
                handleException(streamSegmentName, e);
            }
            return v;
        }, executor);
    }, streamSegmentName);
}
Also used : lombok.val(lombok.val) Arrays(java.util.Arrays) Storage(io.pravega.segmentstore.storage.Storage) ScheduledFuture(java.util.concurrent.ScheduledFuture) StreamSegmentInformation(io.pravega.segmentstore.contracts.StreamSegmentInformation) StreamSegmentNotExistsException(io.pravega.segmentstore.contracts.StreamSegmentNotExistsException) StorageNotPrimaryException(io.pravega.segmentstore.storage.StorageNotPrimaryException) SLTS_STORAGE_USED_PERCENTAGE(io.pravega.shared.MetricsNames.SLTS_STORAGE_USED_PERCENTAGE) StorageFullException(io.pravega.segmentstore.storage.StorageFullException) ImmutableDate(io.pravega.common.util.ImmutableDate) SegmentProperties(io.pravega.segmentstore.contracts.SegmentProperties) StreamSegmentSealedException(io.pravega.segmentstore.contracts.StreamSegmentSealedException) SegmentHandle(io.pravega.segmentstore.storage.SegmentHandle) STORAGE_METADATA_SIZE(io.pravega.shared.MetricsNames.STORAGE_METADATA_SIZE) Duration(java.time.Duration) MetadataTransaction(io.pravega.segmentstore.storage.metadata.MetadataTransaction) ChunkMetadata(io.pravega.segmentstore.storage.metadata.ChunkMetadata) StorageMetadataWritesFencedOutException(io.pravega.segmentstore.storage.metadata.StorageMetadataWritesFencedOutException) CompletionException(java.util.concurrent.CompletionException) GuardedBy(javax.annotation.concurrent.GuardedBy) StatusFlags(io.pravega.segmentstore.storage.metadata.StatusFlags) Slf4j(lombok.extern.slf4j.Slf4j) List(java.util.List) StreamSegmentExistsException(io.pravega.segmentstore.contracts.StreamSegmentExistsException) ConcurrentModificationException(java.util.ConcurrentModificationException) Futures(io.pravega.common.concurrent.Futures) Getter(lombok.Getter) SegmentRollingPolicy(io.pravega.segmentstore.storage.SegmentRollingPolicy) SLTS_STORAGE_USED_BYTES(io.pravega.shared.MetricsNames.SLTS_STORAGE_USED_BYTES) Exceptions(io.pravega.common.Exceptions) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) Callable(java.util.concurrent.Callable) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) SLTS_DELETE_LATENCY(io.pravega.segmentstore.storage.chunklayer.ChunkStorageMetrics.SLTS_DELETE_LATENCY) HashSet(java.util.HashSet) SegmentMetadata(io.pravega.segmentstore.storage.metadata.SegmentMetadata) MultiKeySequentialProcessor(io.pravega.common.concurrent.MultiKeySequentialProcessor) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) SLTS_CREATE_LATENCY(io.pravega.segmentstore.storage.chunklayer.ChunkStorageMetrics.SLTS_CREATE_LATENCY) SLTS_DELETE_COUNT(io.pravega.segmentstore.storage.chunklayer.ChunkStorageMetrics.SLTS_DELETE_COUNT) LoggerHelpers(io.pravega.common.LoggerHelpers) NameUtils(io.pravega.shared.NameUtils) Iterator(java.util.Iterator) SLTS_CREATE_COUNT(io.pravega.segmentstore.storage.chunklayer.ChunkStorageMetrics.SLTS_CREATE_COUNT) Executor(java.util.concurrent.Executor) STORAGE_METADATA_NUM_CHUNKS(io.pravega.shared.MetricsNames.STORAGE_METADATA_NUM_CHUNKS) INTERNAL_SCOPE_PREFIX(io.pravega.shared.NameUtils.INTERNAL_SCOPE_PREFIX) lombok.val(lombok.val) Timer(io.pravega.common.Timer) Beta(com.google.common.annotations.Beta) TimeUnit(java.util.concurrent.TimeUnit) AtomicLong(java.util.concurrent.atomic.AtomicLong) ChunkMetadataStore(io.pravega.segmentstore.storage.metadata.ChunkMetadataStore) ReadIndexBlockMetadata(io.pravega.segmentstore.storage.metadata.ReadIndexBlockMetadata) Preconditions(com.google.common.base.Preconditions) InputStream(java.io.InputStream) StreamSegmentExistsException(io.pravega.segmentstore.contracts.StreamSegmentExistsException) Timer(io.pravega.common.Timer) CompletionException(java.util.concurrent.CompletionException) Duration(java.time.Duration)

Aggregations

SegmentRollingPolicy (io.pravega.segmentstore.storage.SegmentRollingPolicy)50 Cleanup (lombok.Cleanup)46 lombok.val (lombok.val)45 Test (org.junit.Test)39 ByteArrayInputStream (java.io.ByteArrayInputStream)32 InMemoryTaskQueueManager (io.pravega.segmentstore.storage.mocks.InMemoryTaskQueueManager)21 ChunkMetadataStore (io.pravega.segmentstore.storage.metadata.ChunkMetadataStore)19 InMemoryChunkStorage (io.pravega.segmentstore.storage.mocks.InMemoryChunkStorage)17 InMemorySnapshotInfoStore (io.pravega.segmentstore.storage.mocks.InMemorySnapshotInfoStore)11 HashSet (java.util.HashSet)10 ArrayList (java.util.ArrayList)9 CompletableFuture (java.util.concurrent.CompletableFuture)9 StreamSegmentNotExistsException (io.pravega.segmentstore.contracts.StreamSegmentNotExistsException)7 InMemoryMetadataStore (io.pravega.segmentstore.storage.mocks.InMemoryMetadataStore)7 NoOpChunkStorage (io.pravega.segmentstore.storage.noop.NoOpChunkStorage)7 ScheduledExecutorService (java.util.concurrent.ScheduledExecutorService)7 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)7 SegmentProperties (io.pravega.segmentstore.contracts.SegmentProperties)6 StorageFullException (io.pravega.segmentstore.storage.StorageFullException)6 StorageNotPrimaryException (io.pravega.segmentstore.storage.StorageNotPrimaryException)6