Search in sources :

Example 1 with LocalRecoveryConfig

use of org.apache.flink.runtime.state.LocalRecoveryConfig in project flink by apache.

the class LocalStateForwardingTest method testReportingFromTaskStateManagerToResponderAndTaskLocalStateStore.

/**
 * This tests that state that was reported to the {@link
 * org.apache.flink.runtime.state.TaskStateManager} is also reported to {@link
 * org.apache.flink.runtime.taskmanager.CheckpointResponder} and {@link
 * TaskLocalStateStoreImpl}.
 */
@Test
public void testReportingFromTaskStateManagerToResponderAndTaskLocalStateStore() throws Exception {
    final JobID jobID = new JobID();
    final AllocationID allocationID = new AllocationID();
    final ExecutionAttemptID executionAttemptID = new ExecutionAttemptID();
    final CheckpointMetaData checkpointMetaData = new CheckpointMetaData(42L, 4711L);
    final CheckpointMetrics checkpointMetrics = new CheckpointMetrics();
    final int subtaskIdx = 42;
    JobVertexID jobVertexID = new JobVertexID();
    TaskStateSnapshot jmSnapshot = new TaskStateSnapshot();
    TaskStateSnapshot tmSnapshot = new TaskStateSnapshot();
    final AtomicBoolean jmReported = new AtomicBoolean(false);
    final AtomicBoolean tmReported = new AtomicBoolean(false);
    TestCheckpointResponder checkpointResponder = new TestCheckpointResponder() {

        @Override
        public void acknowledgeCheckpoint(JobID lJobID, ExecutionAttemptID lExecutionAttemptID, long lCheckpointId, CheckpointMetrics lCheckpointMetrics, TaskStateSnapshot lSubtaskState) {
            Assert.assertEquals(jobID, lJobID);
            Assert.assertEquals(executionAttemptID, lExecutionAttemptID);
            Assert.assertEquals(checkpointMetaData.getCheckpointId(), lCheckpointId);
            Assert.assertEquals(checkpointMetrics, lCheckpointMetrics);
            jmReported.set(true);
        }
    };
    Executor executor = Executors.directExecutor();
    LocalRecoveryDirectoryProviderImpl directoryProvider = new LocalRecoveryDirectoryProviderImpl(temporaryFolder.newFolder(), jobID, jobVertexID, subtaskIdx);
    LocalRecoveryConfig localRecoveryConfig = new LocalRecoveryConfig(directoryProvider);
    TaskLocalStateStore taskLocalStateStore = new TaskLocalStateStoreImpl(jobID, allocationID, jobVertexID, subtaskIdx, localRecoveryConfig, executor) {

        @Override
        public void storeLocalState(@Nonnegative long checkpointId, @Nullable TaskStateSnapshot localState) {
            Assert.assertEquals(tmSnapshot, localState);
            tmReported.set(true);
        }
    };
    StateChangelogStorage<?> stateChangelogStorage = new InMemoryStateChangelogStorage();
    TaskStateManagerImpl taskStateManager = new TaskStateManagerImpl(jobID, executionAttemptID, taskLocalStateStore, stateChangelogStorage, null, checkpointResponder);
    taskStateManager.reportTaskStateSnapshots(checkpointMetaData, checkpointMetrics, jmSnapshot, tmSnapshot);
    Assert.assertTrue("Reporting for JM state was not called.", jmReported.get());
    Assert.assertTrue("Reporting for TM state was not called.", tmReported.get());
}
Also used : TaskStateManagerImpl(org.apache.flink.runtime.state.TaskStateManagerImpl) ExecutionAttemptID(org.apache.flink.runtime.executiongraph.ExecutionAttemptID) TaskLocalStateStore(org.apache.flink.runtime.state.TaskLocalStateStore) AllocationID(org.apache.flink.runtime.clusterframework.types.AllocationID) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) CheckpointMetrics(org.apache.flink.runtime.checkpoint.CheckpointMetrics) LocalRecoveryDirectoryProviderImpl(org.apache.flink.runtime.state.LocalRecoveryDirectoryProviderImpl) LocalRecoveryConfig(org.apache.flink.runtime.state.LocalRecoveryConfig) CheckpointMetaData(org.apache.flink.runtime.checkpoint.CheckpointMetaData) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) TaskStateSnapshot(org.apache.flink.runtime.checkpoint.TaskStateSnapshot) Executor(java.util.concurrent.Executor) InMemoryStateChangelogStorage(org.apache.flink.runtime.state.changelog.inmemory.InMemoryStateChangelogStorage) TaskLocalStateStoreImpl(org.apache.flink.runtime.state.TaskLocalStateStoreImpl) Nonnegative(javax.annotation.Nonnegative) TestCheckpointResponder(org.apache.flink.runtime.taskmanager.TestCheckpointResponder) JobID(org.apache.flink.api.common.JobID) Nullable(javax.annotation.Nullable) Test(org.junit.Test)

Example 2 with LocalRecoveryConfig

use of org.apache.flink.runtime.state.LocalRecoveryConfig in project flink by apache.

the class StateBackendBenchmarkUtils method createRocksDBKeyedStateBackend.

private static RocksDBKeyedStateBackend<Long> createRocksDBKeyedStateBackend(File rootDir) throws IOException {
    File recoveryBaseDir = prepareDirectory(recoveryDirName, rootDir);
    File dbPathFile = prepareDirectory(dbDirName, rootDir);
    ExecutionConfig executionConfig = new ExecutionConfig();
    RocksDBResourceContainer resourceContainer = new RocksDBResourceContainer();
    RocksDBKeyedStateBackendBuilder<Long> builder = new RocksDBKeyedStateBackendBuilder<>("Test", Thread.currentThread().getContextClassLoader(), dbPathFile, resourceContainer, stateName -> resourceContainer.getColumnOptions(), null, LongSerializer.INSTANCE, 2, new KeyGroupRange(0, 1), executionConfig, new LocalRecoveryConfig(null), EmbeddedRocksDBStateBackend.PriorityQueueStateType.ROCKSDB, TtlTimeProvider.DEFAULT, LatencyTrackingStateConfig.disabled(), new UnregisteredMetricsGroup(), Collections.emptyList(), AbstractStateBackend.getCompressionDecorator(executionConfig), new CloseableRegistry());
    try {
        return builder.build();
    } catch (Exception e) {
        IOUtils.closeQuietly(resourceContainer);
        throw e;
    }
}
Also used : RocksDBResourceContainer(org.apache.flink.contrib.streaming.state.RocksDBResourceContainer) UnregisteredMetricsGroup(org.apache.flink.metrics.groups.UnregisteredMetricsGroup) RocksDBKeyedStateBackendBuilder(org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) LocalRecoveryConfig(org.apache.flink.runtime.state.LocalRecoveryConfig) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) File(java.io.File) RocksDBException(org.rocksdb.RocksDBException) IOException(java.io.IOException)

Example 3 with LocalRecoveryConfig

use of org.apache.flink.runtime.state.LocalRecoveryConfig in project flink by apache.

the class StateBackendBenchmarkUtils method createHeapKeyedStateBackend.

private static HeapKeyedStateBackend<Long> createHeapKeyedStateBackend(File rootDir) throws IOException {
    File recoveryBaseDir = prepareDirectory(recoveryDirName, rootDir);
    KeyGroupRange keyGroupRange = new KeyGroupRange(0, 1);
    int numberOfKeyGroups = keyGroupRange.getNumberOfKeyGroups();
    ExecutionConfig executionConfig = new ExecutionConfig();
    HeapPriorityQueueSetFactory priorityQueueSetFactory = new HeapPriorityQueueSetFactory(keyGroupRange, numberOfKeyGroups, 128);
    HeapKeyedStateBackendBuilder<Long> backendBuilder = new HeapKeyedStateBackendBuilder<>(null, new LongSerializer(), Thread.currentThread().getContextClassLoader(), numberOfKeyGroups, keyGroupRange, executionConfig, TtlTimeProvider.DEFAULT, LatencyTrackingStateConfig.disabled(), Collections.emptyList(), AbstractStateBackend.getCompressionDecorator(executionConfig), new LocalRecoveryConfig(null), priorityQueueSetFactory, false, new CloseableRegistry());
    return backendBuilder.build();
}
Also used : LongSerializer(org.apache.flink.api.common.typeutils.base.LongSerializer) HeapPriorityQueueSetFactory(org.apache.flink.runtime.state.heap.HeapPriorityQueueSetFactory) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) LocalRecoveryConfig(org.apache.flink.runtime.state.LocalRecoveryConfig) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) File(java.io.File) HeapKeyedStateBackendBuilder(org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder)

Example 4 with LocalRecoveryConfig

use of org.apache.flink.runtime.state.LocalRecoveryConfig in project flink by apache.

the class FsStateBackend method createKeyedStateBackend.

// ------------------------------------------------------------------------
// state holding structures
// ------------------------------------------------------------------------
@Override
public <K> AbstractKeyedStateBackend<K> createKeyedStateBackend(Environment env, JobID jobID, String operatorIdentifier, TypeSerializer<K> keySerializer, int numberOfKeyGroups, KeyGroupRange keyGroupRange, TaskKvStateRegistry kvStateRegistry, TtlTimeProvider ttlTimeProvider, MetricGroup metricGroup, @Nonnull Collection<KeyedStateHandle> stateHandles, CloseableRegistry cancelStreamRegistry) throws BackendBuildingException {
    TaskStateManager taskStateManager = env.getTaskStateManager();
    LocalRecoveryConfig localRecoveryConfig = taskStateManager.createLocalRecoveryConfig();
    HeapPriorityQueueSetFactory priorityQueueSetFactory = new HeapPriorityQueueSetFactory(keyGroupRange, numberOfKeyGroups, 128);
    LatencyTrackingStateConfig latencyTrackingStateConfig = latencyTrackingConfigBuilder.setMetricGroup(metricGroup).build();
    return new HeapKeyedStateBackendBuilder<>(kvStateRegistry, keySerializer, env.getUserCodeClassLoader().asClassLoader(), numberOfKeyGroups, keyGroupRange, env.getExecutionConfig(), ttlTimeProvider, latencyTrackingStateConfig, stateHandles, AbstractStateBackend.getCompressionDecorator(env.getExecutionConfig()), localRecoveryConfig, priorityQueueSetFactory, isUsingAsynchronousSnapshots(), cancelStreamRegistry).build();
}
Also used : HeapPriorityQueueSetFactory(org.apache.flink.runtime.state.heap.HeapPriorityQueueSetFactory) LatencyTrackingStateConfig(org.apache.flink.runtime.state.metrics.LatencyTrackingStateConfig) LocalRecoveryConfig(org.apache.flink.runtime.state.LocalRecoveryConfig) TaskStateManager(org.apache.flink.runtime.state.TaskStateManager)

Example 5 with LocalRecoveryConfig

use of org.apache.flink.runtime.state.LocalRecoveryConfig in project flink by apache.

the class HashMapStateBackend method createKeyedStateBackend.

@Override
public <K> AbstractKeyedStateBackend<K> createKeyedStateBackend(Environment env, JobID jobID, String operatorIdentifier, TypeSerializer<K> keySerializer, int numberOfKeyGroups, KeyGroupRange keyGroupRange, TaskKvStateRegistry kvStateRegistry, TtlTimeProvider ttlTimeProvider, MetricGroup metricGroup, @Nonnull Collection<KeyedStateHandle> stateHandles, CloseableRegistry cancelStreamRegistry) throws IOException {
    TaskStateManager taskStateManager = env.getTaskStateManager();
    LocalRecoveryConfig localRecoveryConfig = taskStateManager.createLocalRecoveryConfig();
    HeapPriorityQueueSetFactory priorityQueueSetFactory = new HeapPriorityQueueSetFactory(keyGroupRange, numberOfKeyGroups, 128);
    LatencyTrackingStateConfig latencyTrackingStateConfig = latencyTrackingConfigBuilder.setMetricGroup(metricGroup).build();
    return new HeapKeyedStateBackendBuilder<>(kvStateRegistry, keySerializer, env.getUserCodeClassLoader().asClassLoader(), numberOfKeyGroups, keyGroupRange, env.getExecutionConfig(), ttlTimeProvider, latencyTrackingStateConfig, stateHandles, getCompressionDecorator(env.getExecutionConfig()), localRecoveryConfig, priorityQueueSetFactory, true, cancelStreamRegistry).build();
}
Also used : HeapPriorityQueueSetFactory(org.apache.flink.runtime.state.heap.HeapPriorityQueueSetFactory) LatencyTrackingStateConfig(org.apache.flink.runtime.state.metrics.LatencyTrackingStateConfig) LocalRecoveryConfig(org.apache.flink.runtime.state.LocalRecoveryConfig) TaskStateManager(org.apache.flink.runtime.state.TaskStateManager)

Aggregations

LocalRecoveryConfig (org.apache.flink.runtime.state.LocalRecoveryConfig)7 File (java.io.File)4 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)3 HeapPriorityQueueSetFactory (org.apache.flink.runtime.state.heap.HeapPriorityQueueSetFactory)3 LatencyTrackingStateConfig (org.apache.flink.runtime.state.metrics.LatencyTrackingStateConfig)3 IOException (java.io.IOException)2 JobID (org.apache.flink.api.common.JobID)2 CloseableRegistry (org.apache.flink.core.fs.CloseableRegistry)2 JobVertexID (org.apache.flink.runtime.jobgraph.JobVertexID)2 KeyGroupRange (org.apache.flink.runtime.state.KeyGroupRange)2 LocalRecoveryDirectoryProviderImpl (org.apache.flink.runtime.state.LocalRecoveryDirectoryProviderImpl)2 TaskStateManager (org.apache.flink.runtime.state.TaskStateManager)2 InputStream (java.io.InputStream)1 Arrays (java.util.Arrays)1 BitSet (java.util.BitSet)1 Collection (java.util.Collection)1 Executor (java.util.concurrent.Executor)1 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)1 Nonnegative (javax.annotation.Nonnegative)1 Nullable (javax.annotation.Nullable)1