Search in sources :

Example 1 with ByteSerde

use of org.apache.samza.serializers.ByteSerde in project samza by apache.

the class StorageRecovery method getTaskStorageManagers.

/**
   * create one TaskStorageManager for each task. Add all of them to the
   * List<TaskStorageManager>
   */
@SuppressWarnings({ "unchecked", "rawtypes" })
private void getTaskStorageManagers() {
    StreamMetadataCache streamMetadataCache = new StreamMetadataCache(Util.javaMapAsScalaMap(systemAdmins), 5000, SystemClock.instance());
    for (ContainerModel containerModel : containers.values()) {
        HashMap<String, StorageEngine> taskStores = new HashMap<String, StorageEngine>();
        SamzaContainerContext containerContext = new SamzaContainerContext(containerModel.getProcessorId(), jobConfig, containerModel.getTasks().keySet());
        for (TaskModel taskModel : containerModel.getTasks().values()) {
            HashMap<String, SystemConsumer> storeConsumers = getStoreConsumers();
            for (Entry<String, StorageEngineFactory<?, ?>> entry : storageEngineFactories.entrySet()) {
                String storeName = entry.getKey();
                if (changeLogSystemStreams.containsKey(storeName)) {
                    SystemStreamPartition changeLogSystemStreamPartition = new SystemStreamPartition(changeLogSystemStreams.get(storeName), taskModel.getChangelogPartition());
                    File storePartitionDir = TaskStorageManager.getStorePartitionDir(storeBaseDir, storeName, taskModel.getTaskName());
                    log.info("Got storage engine directory: " + storePartitionDir);
                    StorageEngine storageEngine = (entry.getValue()).getStorageEngine(storeName, storePartitionDir, (Serde) new ByteSerde(), (Serde) new ByteSerde(), null, new MetricsRegistryMap(), changeLogSystemStreamPartition, containerContext);
                    taskStores.put(storeName, storageEngine);
                }
            }
            TaskStorageManager taskStorageManager = new TaskStorageManager(taskModel.getTaskName(), Util.javaMapAsScalaMap(taskStores), Util.javaMapAsScalaMap(storeConsumers), Util.javaMapAsScalaMap(changeLogSystemStreams), maxPartitionNumber, streamMetadataCache, storeBaseDir, storeBaseDir, taskModel.getChangelogPartition(), Util.javaMapAsScalaMap(systemAdmins), new StorageConfig(jobConfig).getChangeLogDeleteRetentionsInMs(), new SystemClock());
            taskStorageManagers.add(taskStorageManager);
        }
    }
}
Also used : StreamMetadataCache(org.apache.samza.system.StreamMetadataCache) SamzaContainerContext(org.apache.samza.container.SamzaContainerContext) SystemConsumer(org.apache.samza.system.SystemConsumer) SystemClock(org.apache.samza.util.SystemClock) HashMap(java.util.HashMap) StorageConfig(org.apache.samza.config.StorageConfig) JavaStorageConfig(org.apache.samza.config.JavaStorageConfig) ContainerModel(org.apache.samza.job.model.ContainerModel) ByteSerde(org.apache.samza.serializers.ByteSerde) MetricsRegistryMap(org.apache.samza.metrics.MetricsRegistryMap) File(java.io.File) TaskModel(org.apache.samza.job.model.TaskModel) SystemStreamPartition(org.apache.samza.system.SystemStreamPartition)

Example 2 with ByteSerde

use of org.apache.samza.serializers.ByteSerde in project samza by apache.

the class TestTimestampedValueSerde method testEmptyValueDeserialization.

@Test
public void testEmptyValueDeserialization() {
    byte[] bytesWithNoValue = new byte[8];
    ByteBuffer.wrap(bytesWithNoValue).putLong(1234L);
    TimestampedValueSerde<byte[]> timestampedValueSerde = new TimestampedValueSerde<>(new ByteSerde());
    TimestampedValue<byte[]> timestampedValue = timestampedValueSerde.fromBytes(bytesWithNoValue);
    assertEquals(1234L, timestampedValue.getTimestamp());
    assertEquals(0, timestampedValue.getValue().length);
}
Also used : ByteSerde(org.apache.samza.serializers.ByteSerde) Test(org.junit.Test)

Aggregations

ByteSerde (org.apache.samza.serializers.ByteSerde)2 File (java.io.File)1 HashMap (java.util.HashMap)1 JavaStorageConfig (org.apache.samza.config.JavaStorageConfig)1 StorageConfig (org.apache.samza.config.StorageConfig)1 SamzaContainerContext (org.apache.samza.container.SamzaContainerContext)1 ContainerModel (org.apache.samza.job.model.ContainerModel)1 TaskModel (org.apache.samza.job.model.TaskModel)1 MetricsRegistryMap (org.apache.samza.metrics.MetricsRegistryMap)1 StreamMetadataCache (org.apache.samza.system.StreamMetadataCache)1 SystemConsumer (org.apache.samza.system.SystemConsumer)1 SystemStreamPartition (org.apache.samza.system.SystemStreamPartition)1 SystemClock (org.apache.samza.util.SystemClock)1 Test (org.junit.Test)1