Search in sources :

Example 1 with S3StorageConfig

use of io.pravega.storage.s3.S3StorageConfig in project pravega by pravega.

the class StorageFactoryTests method testS3StorageFactoryCreator.

private void testS3StorageFactoryCreator(S3StorageConfig config) {
    StorageFactoryCreator factoryCreator = new S3StorageFactoryCreator();
    val expected = new StorageFactoryInfo[] { StorageFactoryInfo.builder().name("S3").storageLayoutType(StorageLayoutType.CHUNKED_STORAGE).build() };
    val factoryInfoList = factoryCreator.getStorageFactories();
    Assert.assertEquals(1, factoryInfoList.length);
    Assert.assertArrayEquals(expected, factoryInfoList);
    // Simple Storage
    ConfigSetup configSetup1 = mock(ConfigSetup.class);
    when(configSetup1.getConfig(any())).thenReturn(ChunkedSegmentStorageConfig.DEFAULT_CONFIG, config);
    val factory1 = factoryCreator.createFactory(expected[0], configSetup1, executorService());
    Assert.assertTrue(factory1 instanceof S3SimpleStorageFactory);
    @Cleanup Storage storage1 = ((S3SimpleStorageFactory) factory1).createStorageAdapter(42, new InMemoryMetadataStore(ChunkedSegmentStorageConfig.DEFAULT_CONFIG, executorService()));
    Assert.assertTrue(storage1 instanceof ChunkedSegmentStorage);
    Assert.assertTrue(((ChunkedSegmentStorage) storage1).getChunkStorage() instanceof S3ChunkStorage);
    AssertExtensions.assertThrows("createStorageAdapter should throw UnsupportedOperationException.", () -> factory1.createStorageAdapter(), ex -> ex instanceof UnsupportedOperationException);
}
Also used : lombok.val(lombok.val) Storage(io.pravega.segmentstore.storage.Storage) ChunkedSegmentStorage(io.pravega.segmentstore.storage.chunklayer.ChunkedSegmentStorage) ExtendedS3ChunkStorage(io.pravega.storage.extendeds3.ExtendedS3ChunkStorage) HDFSChunkStorage(io.pravega.storage.hdfs.HDFSChunkStorage) S3ChunkStorage(io.pravega.storage.s3.S3ChunkStorage) SyncStorage(io.pravega.segmentstore.storage.SyncStorage) FileSystemChunkStorage(io.pravega.storage.filesystem.FileSystemChunkStorage) ExtendedS3SimpleStorageFactory(io.pravega.storage.extendeds3.ExtendedS3SimpleStorageFactory) S3SimpleStorageFactory(io.pravega.storage.s3.S3SimpleStorageFactory) ExtendedS3ChunkStorage(io.pravega.storage.extendeds3.ExtendedS3ChunkStorage) S3ChunkStorage(io.pravega.storage.s3.S3ChunkStorage) InMemoryMetadataStore(io.pravega.segmentstore.storage.mocks.InMemoryMetadataStore) StorageFactoryInfo(io.pravega.segmentstore.storage.StorageFactoryInfo) ChunkedSegmentStorage(io.pravega.segmentstore.storage.chunklayer.ChunkedSegmentStorage) S3StorageFactoryCreator(io.pravega.storage.s3.S3StorageFactoryCreator) HDFSStorageFactoryCreator(io.pravega.storage.hdfs.HDFSStorageFactoryCreator) StorageFactoryCreator(io.pravega.segmentstore.storage.StorageFactoryCreator) ExtendedS3StorageFactoryCreator(io.pravega.storage.extendeds3.ExtendedS3StorageFactoryCreator) FileSystemStorageFactoryCreator(io.pravega.storage.filesystem.FileSystemStorageFactoryCreator) S3StorageFactoryCreator(io.pravega.storage.s3.S3StorageFactoryCreator) ExtendedS3StorageFactoryCreator(io.pravega.storage.extendeds3.ExtendedS3StorageFactoryCreator) Cleanup(lombok.Cleanup) ConfigSetup(io.pravega.segmentstore.storage.ConfigSetup)

Example 2 with S3StorageConfig

use of io.pravega.storage.s3.S3StorageConfig in project pravega by pravega.

the class S3IntegrationTest method createBuilder.

// endregion
// region StreamSegmentStoreTestBase Implementation
@Override
protected ServiceBuilder createBuilder(ServiceBuilderConfig.Builder configBuilder, int instanceId, boolean useChunkedSegmentStorage) {
    Preconditions.checkState(useChunkedSegmentStorage);
    ServiceBuilderConfig builderConfig = getBuilderConfig(configBuilder, instanceId);
    return ServiceBuilder.newInMemoryBuilder(builderConfig).withStorageFactory(setup -> new LocalS3SimpleStorageFactory(setup.getConfig(S3StorageConfig::builder), setup.getStorageExecutor())).withDataLogFactory(setup -> new BookKeeperLogFactory(setup.getConfig(BookKeeperConfig::builder), getBookkeeper().getZkClient(), setup.getCoreExecutor()));
}
Also used : S3Mock(io.pravega.storage.s3.S3Mock) Storage(io.pravega.segmentstore.storage.Storage) Getter(lombok.Getter) UUID(java.util.UUID) ChunkedSegmentStorageConfig(io.pravega.segmentstore.storage.chunklayer.ChunkedSegmentStorageConfig) S3StorageConfig(io.pravega.storage.s3.S3StorageConfig) ServiceBuilderConfig(io.pravega.segmentstore.server.store.ServiceBuilderConfig) BookKeeperConfig(io.pravega.segmentstore.storage.impl.bookkeeper.BookKeeperConfig) S3ChunkStorage(io.pravega.storage.s3.S3ChunkStorage) SimpleStorageFactory(io.pravega.segmentstore.storage.SimpleStorageFactory) ServiceBuilder(io.pravega.segmentstore.server.store.ServiceBuilder) S3ClientMock(io.pravega.storage.s3.S3ClientMock) ChunkMetadataStore(io.pravega.segmentstore.storage.metadata.ChunkMetadataStore) After(org.junit.After) Duration(java.time.Duration) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) Preconditions(com.google.common.base.Preconditions) URI(java.net.URI) BookKeeperLogFactory(io.pravega.segmentstore.storage.impl.bookkeeper.BookKeeperLogFactory) ChunkedSegmentStorage(io.pravega.segmentstore.storage.chunklayer.ChunkedSegmentStorage) Before(org.junit.Before) BookKeeperConfig(io.pravega.segmentstore.storage.impl.bookkeeper.BookKeeperConfig) ServiceBuilderConfig(io.pravega.segmentstore.server.store.ServiceBuilderConfig) BookKeeperLogFactory(io.pravega.segmentstore.storage.impl.bookkeeper.BookKeeperLogFactory)

Aggregations

Storage (io.pravega.segmentstore.storage.Storage)2 ChunkedSegmentStorage (io.pravega.segmentstore.storage.chunklayer.ChunkedSegmentStorage)2 S3ChunkStorage (io.pravega.storage.s3.S3ChunkStorage)2 Preconditions (com.google.common.base.Preconditions)1 ServiceBuilder (io.pravega.segmentstore.server.store.ServiceBuilder)1 ServiceBuilderConfig (io.pravega.segmentstore.server.store.ServiceBuilderConfig)1 ConfigSetup (io.pravega.segmentstore.storage.ConfigSetup)1 SimpleStorageFactory (io.pravega.segmentstore.storage.SimpleStorageFactory)1 StorageFactoryCreator (io.pravega.segmentstore.storage.StorageFactoryCreator)1 StorageFactoryInfo (io.pravega.segmentstore.storage.StorageFactoryInfo)1 SyncStorage (io.pravega.segmentstore.storage.SyncStorage)1 ChunkedSegmentStorageConfig (io.pravega.segmentstore.storage.chunklayer.ChunkedSegmentStorageConfig)1 BookKeeperConfig (io.pravega.segmentstore.storage.impl.bookkeeper.BookKeeperConfig)1 BookKeeperLogFactory (io.pravega.segmentstore.storage.impl.bookkeeper.BookKeeperLogFactory)1 ChunkMetadataStore (io.pravega.segmentstore.storage.metadata.ChunkMetadataStore)1 InMemoryMetadataStore (io.pravega.segmentstore.storage.mocks.InMemoryMetadataStore)1 ExtendedS3ChunkStorage (io.pravega.storage.extendeds3.ExtendedS3ChunkStorage)1 ExtendedS3SimpleStorageFactory (io.pravega.storage.extendeds3.ExtendedS3SimpleStorageFactory)1 ExtendedS3StorageFactoryCreator (io.pravega.storage.extendeds3.ExtendedS3StorageFactoryCreator)1 FileSystemChunkStorage (io.pravega.storage.filesystem.FileSystemChunkStorage)1