Search in sources :

Example 1 with PlaceholderStreamStateHandle

use of org.apache.flink.runtime.state.PlaceholderStreamStateHandle in project flink by apache.

the class SchedulerUtilsTest method testSharedStateRegistration.

/**
 * Check that a {@link SharedStateRegistryFactory} used by {@link SchedulerUtils} registers
 * shared checkpoint state on restore.
 */
@Test
public void testSharedStateRegistration() throws Exception {
    UUID backendId = UUID.randomUUID();
    StateHandleID key = new StateHandleID("k0");
    StreamStateHandle handle = new ByteStreamStateHandle("h0", new byte[] { 1, 2, 3 });
    CheckpointRecoveryFactory recoveryFactory = buildRecoveryFactory(buildCheckpoint(buildIncrementalHandle(key, handle, backendId)));
    CompletedCheckpointStore checkpointStore = SchedulerUtils.createCompletedCheckpointStore(new Configuration(), recoveryFactory, Executors.directExecutor(), log, new JobID());
    SharedStateRegistry sharedStateRegistry = checkpointStore.getSharedStateRegistry();
    IncrementalRemoteKeyedStateHandle newHandle = buildIncrementalHandle(key, new PlaceholderStreamStateHandle(1L), backendId);
    newHandle.registerSharedStates(sharedStateRegistry, 1L);
    assertSame(handle, newHandle.getSharedState().get(key));
}
Also used : PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) StreamStateHandle(org.apache.flink.runtime.state.StreamStateHandle) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) ByteStreamStateHandle(org.apache.flink.runtime.state.memory.ByteStreamStateHandle) Configuration(org.apache.flink.configuration.Configuration) StateHandleID(org.apache.flink.runtime.state.StateHandleID) IncrementalRemoteKeyedStateHandle(org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle) ByteStreamStateHandle(org.apache.flink.runtime.state.memory.ByteStreamStateHandle) UUID(java.util.UUID) CheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory) StandaloneCheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.StandaloneCheckpointRecoveryFactory) JobID(org.apache.flink.api.common.JobID) CompletedCheckpointStore(org.apache.flink.runtime.checkpoint.CompletedCheckpointStore) EmbeddedCompletedCheckpointStore(org.apache.flink.runtime.checkpoint.EmbeddedCompletedCheckpointStore) SharedStateRegistry(org.apache.flink.runtime.state.SharedStateRegistry) Test(org.junit.Test)

Example 2 with PlaceholderStreamStateHandle

use of org.apache.flink.runtime.state.PlaceholderStreamStateHandle in project flink by apache.

the class RocksIncrementalSnapshotStrategyTest method testCheckpointIsIncremental.

// Verify the next checkpoint is still incremental after a savepoint completed.
@Test
public void testCheckpointIsIncremental() throws Exception {
    try (CloseableRegistry closeableRegistry = new CloseableRegistry();
        RocksIncrementalSnapshotStrategy checkpointSnapshotStrategy = createSnapshotStrategy(closeableRegistry)) {
        FsCheckpointStreamFactory checkpointStreamFactory = createFsCheckpointStreamFactory();
        // make and notify checkpoint with id 1
        snapshot(1L, checkpointSnapshotStrategy, checkpointStreamFactory, closeableRegistry);
        checkpointSnapshotStrategy.notifyCheckpointComplete(1L);
        // notify savepoint with id 2
        checkpointSnapshotStrategy.notifyCheckpointComplete(2L);
        // make checkpoint with id 3
        IncrementalRemoteKeyedStateHandle incrementalRemoteKeyedStateHandle3 = snapshot(3L, checkpointSnapshotStrategy, checkpointStreamFactory, closeableRegistry);
        // If 3rd checkpoint's placeholderStateHandleCount > 0,it means 3rd checkpoint is
        // incremental.
        Map<StateHandleID, StreamStateHandle> sharedState3 = incrementalRemoteKeyedStateHandle3.getSharedState();
        long placeholderStateHandleCount = sharedState3.entrySet().stream().filter(e -> e.getValue() instanceof PlaceholderStreamStateHandle).count();
        Assert.assertTrue(placeholderStateHandleCount > 0);
    }
}
Also used : RocksDBResource(org.apache.flink.contrib.streaming.state.RocksDBResource) StateHandleID(org.apache.flink.runtime.state.StateHandleID) IncrementalRemoteKeyedStateHandle(org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle) ResourceGuard(org.apache.flink.util.ResourceGuard) ArrayList(java.util.ArrayList) LinkedHashMap(java.util.LinkedHashMap) IntSerializer(org.apache.flink.api.common.typeutils.base.IntSerializer) RocksDBOptions(org.apache.flink.contrib.streaming.state.RocksDBOptions) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) RocksDB(org.rocksdb.RocksDB) TestLocalRecoveryConfig(org.apache.flink.runtime.state.TestLocalRecoveryConfig) Map(java.util.Map) RocksDBException(org.rocksdb.RocksDBException) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) StateDescriptor(org.apache.flink.api.common.state.StateDescriptor) CompositeKeySerializationUtils(org.apache.flink.runtime.state.CompositeKeySerializationUtils) Test(org.junit.Test) IOException(java.io.IOException) CheckpointOptions(org.apache.flink.runtime.checkpoint.CheckpointOptions) UUID(java.util.UUID) ArrayListSerializer(org.apache.flink.runtime.state.ArrayListSerializer) StreamStateHandle(org.apache.flink.runtime.state.StreamStateHandle) File(java.io.File) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) Rule(org.junit.Rule) TreeMap(java.util.TreeMap) FsCheckpointStreamFactory(org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory) ColumnFamilyHandle(org.rocksdb.ColumnFamilyHandle) RegisteredKeyValueStateBackendMetaInfo(org.apache.flink.runtime.state.RegisteredKeyValueStateBackendMetaInfo) RocksDBKeyedStateBackend(org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend) Assert(org.junit.Assert) TemporaryFolder(org.junit.rules.TemporaryFolder) SortedMap(java.util.SortedMap) LocalFileSystem.getSharedInstance(org.apache.flink.core.fs.local.LocalFileSystem.getSharedInstance) RocksDBStateUploader(org.apache.flink.contrib.streaming.state.RocksDBStateUploader) Path.fromLocalFile(org.apache.flink.core.fs.Path.fromLocalFile) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) StreamStateHandle(org.apache.flink.runtime.state.StreamStateHandle) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) StateHandleID(org.apache.flink.runtime.state.StateHandleID) FsCheckpointStreamFactory(org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory) IncrementalRemoteKeyedStateHandle(org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) Test(org.junit.Test)

Example 3 with PlaceholderStreamStateHandle

use of org.apache.flink.runtime.state.PlaceholderStreamStateHandle in project flink by apache.

the class CheckpointCoordinatorTest method testSharedStateRegistrationOnRestore.

@Test
public void testSharedStateRegistrationOnRestore() throws Exception {
    JobVertexID jobVertexID1 = new JobVertexID();
    int parallelism1 = 2;
    int maxParallelism1 = 4;
    ExecutionGraph graph = new CheckpointCoordinatorTestingUtils.CheckpointExecutionGraphBuilder().addJobVertex(jobVertexID1, parallelism1, maxParallelism1).build();
    ExecutionJobVertex jobVertex1 = graph.getJobVertex(jobVertexID1);
    List<CompletedCheckpoint> checkpoints = Collections.emptyList();
    SharedStateRegistry firstInstance = SharedStateRegistry.DEFAULT_FACTORY.create(org.apache.flink.util.concurrent.Executors.directExecutor(), checkpoints);
    final EmbeddedCompletedCheckpointStore store = new EmbeddedCompletedCheckpointStore(10, checkpoints, firstInstance);
    // set up the coordinator and validate the initial state
    final CheckpointCoordinatorBuilder coordinatorBuilder = new CheckpointCoordinatorBuilder().setExecutionGraph(graph).setTimer(manuallyTriggeredScheduledExecutor);
    final CheckpointCoordinator coordinator = coordinatorBuilder.setCompletedCheckpointStore(store).build();
    final int numCheckpoints = 3;
    List<KeyGroupRange> keyGroupPartitions1 = StateAssignmentOperation.createKeyGroupPartitions(maxParallelism1, parallelism1);
    for (int i = 0; i < numCheckpoints; ++i) {
        performIncrementalCheckpoint(graph.getJobID(), coordinator, jobVertex1, keyGroupPartitions1, i);
    }
    List<CompletedCheckpoint> completedCheckpoints = coordinator.getSuccessfulCheckpoints();
    assertEquals(numCheckpoints, completedCheckpoints.size());
    int sharedHandleCount = 0;
    List<Map<StateHandleID, StreamStateHandle>> sharedHandlesByCheckpoint = new ArrayList<>(numCheckpoints);
    for (int i = 0; i < numCheckpoints; ++i) {
        sharedHandlesByCheckpoint.add(new HashMap<>(2));
    }
    int cp = 0;
    for (CompletedCheckpoint completedCheckpoint : completedCheckpoints) {
        for (OperatorState taskState : completedCheckpoint.getOperatorStates().values()) {
            for (OperatorSubtaskState subtaskState : taskState.getStates()) {
                for (KeyedStateHandle keyedStateHandle : subtaskState.getManagedKeyedState()) {
                    // test we are once registered with the current registry
                    verify(keyedStateHandle, times(1)).registerSharedStates(firstInstance, completedCheckpoint.getCheckpointID());
                    IncrementalRemoteKeyedStateHandle incrementalKeyedStateHandle = (IncrementalRemoteKeyedStateHandle) keyedStateHandle;
                    sharedHandlesByCheckpoint.get(cp).putAll(incrementalKeyedStateHandle.getSharedState());
                    for (StreamStateHandle streamStateHandle : incrementalKeyedStateHandle.getSharedState().values()) {
                        assertTrue(!(streamStateHandle instanceof PlaceholderStreamStateHandle));
                        verify(streamStateHandle, never()).discardState();
                        ++sharedHandleCount;
                    }
                    for (StreamStateHandle streamStateHandle : incrementalKeyedStateHandle.getPrivateState().values()) {
                        verify(streamStateHandle, never()).discardState();
                    }
                    verify(incrementalKeyedStateHandle.getMetaStateHandle(), never()).discardState();
                }
                verify(subtaskState, never()).discardState();
            }
        }
        ++cp;
    }
    // 2 (parallelism) x (1 (CP0) + 2 (CP1) + 2 (CP2)) = 10
    assertEquals(10, sharedHandleCount);
    // discard CP0
    store.removeOldestCheckpoint();
    // CP1
    for (Map<StateHandleID, StreamStateHandle> cpList : sharedHandlesByCheckpoint) {
        for (StreamStateHandle streamStateHandle : cpList.values()) {
            verify(streamStateHandle, never()).discardState();
        }
    }
    // shutdown the store
    store.shutdown(JobStatus.SUSPENDED, new CheckpointsCleaner());
    // restore the store
    Set<ExecutionJobVertex> tasks = new HashSet<>();
    tasks.add(jobVertex1);
    assertEquals(JobStatus.SUSPENDED, store.getShutdownStatus().orElse(null));
    SharedStateRegistry secondInstance = SharedStateRegistry.DEFAULT_FACTORY.create(org.apache.flink.util.concurrent.Executors.directExecutor(), store.getAllCheckpoints());
    final EmbeddedCompletedCheckpointStore secondStore = new EmbeddedCompletedCheckpointStore(10, store.getAllCheckpoints(), secondInstance);
    final CheckpointCoordinator secondCoordinator = coordinatorBuilder.setCompletedCheckpointStore(secondStore).build();
    assertTrue(secondCoordinator.restoreLatestCheckpointedStateToAll(tasks, false));
    // validate that all shared states are registered again after the recovery.
    cp = 0;
    for (CompletedCheckpoint completedCheckpoint : completedCheckpoints) {
        for (OperatorState taskState : completedCheckpoint.getOperatorStates().values()) {
            for (OperatorSubtaskState subtaskState : taskState.getStates()) {
                for (KeyedStateHandle keyedStateHandle : subtaskState.getManagedKeyedState()) {
                    VerificationMode verificationMode;
                    // test we are once registered with the new registry
                    if (cp > 0) {
                        verificationMode = times(1);
                    } else {
                        verificationMode = never();
                    }
                    // check that all are registered with the new registry
                    verify(keyedStateHandle, verificationMode).registerSharedStates(secondInstance, completedCheckpoint.getCheckpointID());
                }
            }
        }
        ++cp;
    }
    // discard CP1
    secondStore.removeOldestCheckpoint();
    // we expect that all shared state from CP0 is no longer referenced and discarded. CP2 is
    // still live and also
    // references the state from CP1, so we expect they are not discarded.
    verifyDiscard(sharedHandlesByCheckpoint, cpId -> cpId == 0 ? times(1) : never());
    // discard CP2
    secondStore.removeOldestCheckpoint();
    // still expect shared state not to be discarded because it may be used in later checkpoints
    verifyDiscard(sharedHandlesByCheckpoint, cpId -> cpId == 1 ? never() : atLeast(0));
}
Also used : JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) ArrayList(java.util.ArrayList) IncrementalRemoteKeyedStateHandle(org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle) KeyedStateHandle(org.apache.flink.runtime.state.KeyedStateHandle) SharedStateRegistry(org.apache.flink.runtime.state.SharedStateRegistry) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) StreamStateHandle(org.apache.flink.runtime.state.StreamStateHandle) ByteStreamStateHandle(org.apache.flink.runtime.state.memory.ByteStreamStateHandle) OperatorStreamStateHandle(org.apache.flink.runtime.state.OperatorStreamStateHandle) TestingStreamStateHandle(org.apache.flink.runtime.state.TestingStreamStateHandle) ExecutionJobVertex(org.apache.flink.runtime.executiongraph.ExecutionJobVertex) IncrementalRemoteKeyedStateHandle(org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle) HashSet(java.util.HashSet) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) AcknowledgeCheckpoint(org.apache.flink.runtime.messages.checkpoint.AcknowledgeCheckpoint) DeclineCheckpoint(org.apache.flink.runtime.messages.checkpoint.DeclineCheckpoint) CheckpointCoordinatorBuilder(org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTestingUtils.CheckpointCoordinatorBuilder) VerificationMode(org.mockito.verification.VerificationMode) StateHandleID(org.apache.flink.runtime.state.StateHandleID) ExecutionGraph(org.apache.flink.runtime.executiongraph.ExecutionGraph) Map(java.util.Map) HashMap(java.util.HashMap) Collections.singletonMap(java.util.Collections.singletonMap) Test(org.junit.Test)

Example 4 with PlaceholderStreamStateHandle

use of org.apache.flink.runtime.state.PlaceholderStreamStateHandle in project flink by apache.

the class CheckpointCoordinatorTest method performIncrementalCheckpoint.

private void performIncrementalCheckpoint(JobID jobId, CheckpointCoordinator checkpointCoordinator, ExecutionJobVertex jobVertex1, List<KeyGroupRange> keyGroupPartitions1, int cpSequenceNumber) throws Exception {
    // trigger the checkpoint
    checkpointCoordinator.triggerCheckpoint(false);
    manuallyTriggeredScheduledExecutor.triggerAll();
    assertEquals(1, checkpointCoordinator.getPendingCheckpoints().size());
    long checkpointId = Iterables.getOnlyElement(checkpointCoordinator.getPendingCheckpoints().keySet());
    for (int index = 0; index < jobVertex1.getParallelism(); index++) {
        KeyGroupRange keyGroupRange = keyGroupPartitions1.get(index);
        Map<StateHandleID, StreamStateHandle> privateState = new HashMap<>();
        privateState.put(new StateHandleID("private-1"), spy(new ByteStreamStateHandle("private-1", new byte[] { 'p' })));
        Map<StateHandleID, StreamStateHandle> sharedState = new HashMap<>();
        // let all but the first CP overlap by one shared state.
        if (cpSequenceNumber > 0) {
            sharedState.put(new StateHandleID("shared-" + (cpSequenceNumber - 1)), spy(new PlaceholderStreamStateHandle(1L)));
        }
        sharedState.put(new StateHandleID("shared-" + cpSequenceNumber), spy(new ByteStreamStateHandle("shared-" + cpSequenceNumber + "-" + keyGroupRange, new byte[] { 's' })));
        IncrementalRemoteKeyedStateHandle managedState = spy(new IncrementalRemoteKeyedStateHandle(new UUID(42L, 42L), keyGroupRange, checkpointId, sharedState, privateState, spy(new ByteStreamStateHandle("meta", new byte[] { 'm' }))));
        OperatorSubtaskState operatorSubtaskState = spy(OperatorSubtaskState.builder().setManagedKeyedState(managedState).build());
        Map<OperatorID, OperatorSubtaskState> opStates = new HashMap<>();
        opStates.put(jobVertex1.getOperatorIDs().get(0).getGeneratedOperatorID(), operatorSubtaskState);
        TaskStateSnapshot taskStateSnapshot = new TaskStateSnapshot(opStates);
        AcknowledgeCheckpoint acknowledgeCheckpoint = new AcknowledgeCheckpoint(jobId, jobVertex1.getTaskVertices()[index].getCurrentExecutionAttempt().getAttemptId(), checkpointId, new CheckpointMetrics(), taskStateSnapshot);
        checkpointCoordinator.receiveAcknowledgeMessage(acknowledgeCheckpoint, TASK_MANAGER_LOCATION_INFO);
    }
}
Also used : PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) HashMap(java.util.HashMap) KeyGroupRange(org.apache.flink.runtime.state.KeyGroupRange) ByteStreamStateHandle(org.apache.flink.runtime.state.memory.ByteStreamStateHandle) OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) AcknowledgeCheckpoint(org.apache.flink.runtime.messages.checkpoint.AcknowledgeCheckpoint) DeclineCheckpoint(org.apache.flink.runtime.messages.checkpoint.DeclineCheckpoint) AcknowledgeCheckpoint(org.apache.flink.runtime.messages.checkpoint.AcknowledgeCheckpoint) PlaceholderStreamStateHandle(org.apache.flink.runtime.state.PlaceholderStreamStateHandle) StreamStateHandle(org.apache.flink.runtime.state.StreamStateHandle) ByteStreamStateHandle(org.apache.flink.runtime.state.memory.ByteStreamStateHandle) OperatorStreamStateHandle(org.apache.flink.runtime.state.OperatorStreamStateHandle) TestingStreamStateHandle(org.apache.flink.runtime.state.TestingStreamStateHandle) StateHandleID(org.apache.flink.runtime.state.StateHandleID) IncrementalRemoteKeyedStateHandle(org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle) UUID(java.util.UUID)

Aggregations

IncrementalRemoteKeyedStateHandle (org.apache.flink.runtime.state.IncrementalRemoteKeyedStateHandle)4 PlaceholderStreamStateHandle (org.apache.flink.runtime.state.PlaceholderStreamStateHandle)4 StateHandleID (org.apache.flink.runtime.state.StateHandleID)4 StreamStateHandle (org.apache.flink.runtime.state.StreamStateHandle)4 UUID (java.util.UUID)3 KeyGroupRange (org.apache.flink.runtime.state.KeyGroupRange)3 ByteStreamStateHandle (org.apache.flink.runtime.state.memory.ByteStreamStateHandle)3 Test (org.junit.Test)3 ArrayList (java.util.ArrayList)2 HashMap (java.util.HashMap)2 Map (java.util.Map)2 AcknowledgeCheckpoint (org.apache.flink.runtime.messages.checkpoint.AcknowledgeCheckpoint)2 DeclineCheckpoint (org.apache.flink.runtime.messages.checkpoint.DeclineCheckpoint)2 OperatorStreamStateHandle (org.apache.flink.runtime.state.OperatorStreamStateHandle)2 TestingStreamStateHandle (org.apache.flink.runtime.state.TestingStreamStateHandle)2 File (java.io.File)1 IOException (java.io.IOException)1 Collections.singletonMap (java.util.Collections.singletonMap)1 HashSet (java.util.HashSet)1 LinkedHashMap (java.util.LinkedHashMap)1