Search in sources :

Example 1 with StreamTaskStateInitializerImpl

use of org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl in project flink by apache.

the class StreamOperatorContextBuilder method build.

StreamOperatorStateContext build(Logger logger) throws IOException {
    final Environment environment = new SavepointEnvironment.Builder(ctx, maxParallelism).setConfiguration(configuration).setSubtaskIndex(split.getSplitNumber()).setPrioritizedOperatorSubtaskState(split.getPrioritizedOperatorSubtaskState()).build();
    StateBackend stateBackend;
    try {
        stateBackend = StateBackendLoader.fromApplicationOrConfigOrDefault(applicationStateBackend, TernaryBoolean.FALSE, configuration, ctx.getUserCodeClassLoader(), logger);
    } catch (DynamicCodeLoadingException e) {
        throw new IOException("Failed to load state backend", e);
    }
    StreamTaskStateInitializer initializer = new StreamTaskStateInitializerImpl(environment, stateBackend);
    try {
        return initializer.streamOperatorStateContext(operatorState.getOperatorID(), operatorState.getOperatorID().toString(), new NeverFireProcessingTimeService(), keyContext, keySerializer, registry, ctx.getMetricGroup(), 1.0, false);
    } catch (Exception e) {
        throw new IOException("Failed to restore state backend", e);
    }
}
Also used : StreamTaskStateInitializer(org.apache.flink.streaming.api.operators.StreamTaskStateInitializer) NeverFireProcessingTimeService(org.apache.flink.state.api.runtime.NeverFireProcessingTimeService) DynamicCodeLoadingException(org.apache.flink.util.DynamicCodeLoadingException) StreamTaskStateInitializerImpl(org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl) SavepointEnvironment(org.apache.flink.state.api.runtime.SavepointEnvironment) Environment(org.apache.flink.runtime.execution.Environment) IOException(java.io.IOException) StateBackend(org.apache.flink.runtime.state.StateBackend) IOException(java.io.IOException) DynamicCodeLoadingException(org.apache.flink.util.DynamicCodeLoadingException)

Example 2 with StreamTaskStateInitializerImpl

use of org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl in project flink by apache.

the class SubtaskCheckpointCoordinatorTest method testNotifyCheckpointSubsumed.

@Test
public void testNotifyCheckpointSubsumed() throws Exception {
    TestTaskStateManager stateManager = new TestTaskStateManager();
    MockEnvironment mockEnvironment = MockEnvironment.builder().setTaskStateManager(stateManager).build();
    try (SubtaskCheckpointCoordinatorImpl subtaskCheckpointCoordinator = (SubtaskCheckpointCoordinatorImpl) new MockSubtaskCheckpointCoordinatorBuilder().setEnvironment(mockEnvironment).setUnalignedCheckpointEnabled(true).build()) {
        StreamMap<String, String> streamMap = new StreamMap<>((MapFunction<String, String>) value -> value);
        streamMap.setProcessingTimeService(new TestProcessingTimeService());
        final OperatorChain<String, AbstractStreamOperator<String>> operatorChain = operatorChain(streamMap);
        StreamTaskStateInitializerImpl stateInitializer = new StreamTaskStateInitializerImpl(mockEnvironment, new TestStateBackend());
        operatorChain.initializeStateAndOpenOperators(stateInitializer);
        long checkpointId = 42L;
        subtaskCheckpointCoordinator.getChannelStateWriter().start(checkpointId, CheckpointOptions.forCheckpointWithDefaultLocation());
        subtaskCheckpointCoordinator.checkpointState(new CheckpointMetaData(checkpointId, System.currentTimeMillis()), CheckpointOptions.forCheckpointWithDefaultLocation(), new CheckpointMetricsBuilder(), operatorChain, false, () -> false);
        long notifySubsumeCheckpointId = checkpointId + 1L;
        // notify checkpoint aborted before execution.
        subtaskCheckpointCoordinator.notifyCheckpointSubsumed(notifySubsumeCheckpointId, operatorChain, () -> true);
        assertEquals(notifySubsumeCheckpointId, ((TestStateBackend.TestKeyedStateBackend<?>) streamMap.getKeyedStateBackend()).getSubsumeCheckpointId());
    }
}
Also used : StreamTaskStateInitializerImpl(org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl) CheckpointMetricsBuilder(org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder) BufferBuilderTestUtils(org.apache.flink.runtime.io.network.buffer.BufferBuilderTestUtils) SavepointType(org.apache.flink.runtime.checkpoint.SavepointType) ExceptionUtils(org.apache.flink.util.ExceptionUtils) StringSerializer(org.apache.flink.api.common.typeutils.base.StringSerializer) MoreExecutors.newDirectExecutorService(org.apache.flink.shaded.guava30.com.google.common.util.concurrent.MoreExecutors.newDirectExecutorService) StreamElementSerializer(org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer) MapFunction(org.apache.flink.api.common.functions.MapFunction) BasicTypeInfo(org.apache.flink.api.common.typeinfo.BasicTypeInfo) TestTaskStateManager(org.apache.flink.runtime.state.TestTaskStateManager) CloseableRegistry(org.apache.flink.core.fs.CloseableRegistry) CheckpointException(org.apache.flink.runtime.checkpoint.CheckpointException) Assert.fail(org.junit.Assert.fail) WatermarkStatus(org.apache.flink.streaming.runtime.watermarkstatus.WatermarkStatus) CheckpointType(org.apache.flink.runtime.checkpoint.CheckpointType) AbstractEvent(org.apache.flink.runtime.event.AbstractEvent) KeyedStateHandle(org.apache.flink.runtime.state.KeyedStateHandle) MockStreamTaskBuilder(org.apache.flink.streaming.util.MockStreamTaskBuilder) CheckpointOptions(org.apache.flink.runtime.checkpoint.CheckpointOptions) Executors(java.util.concurrent.Executors) CountDownLatch(java.util.concurrent.CountDownLatch) SnapshotType(org.apache.flink.runtime.checkpoint.SnapshotType) TestCheckpointStorageWorkerView(org.apache.flink.runtime.state.TestCheckpointStorageWorkerView) Assert.assertFalse(org.junit.Assert.assertFalse) NonRecordWriter(org.apache.flink.runtime.io.network.api.writer.NonRecordWriter) CancelCheckpointMarker(org.apache.flink.runtime.io.network.api.CancelCheckpointMarker) OperatorID(org.apache.flink.runtime.jobgraph.OperatorID) DummyEnvironment(org.apache.flink.runtime.operators.testutils.DummyEnvironment) OneInputStreamOperator(org.apache.flink.streaming.api.operators.OneInputStreamOperator) RecordOrEventCollectingResultPartitionWriter(org.apache.flink.runtime.io.network.api.writer.RecordOrEventCollectingResultPartitionWriter) OneShotLatch(org.apache.flink.core.testutils.OneShotLatch) StreamConfig(org.apache.flink.streaming.api.graph.StreamConfig) CHECKPOINT(org.apache.flink.runtime.checkpoint.CheckpointType.CHECKPOINT) SavepointFormatType(org.apache.flink.core.execution.SavepointFormatType) CheckpointMetaData(org.apache.flink.runtime.checkpoint.CheckpointMetaData) Watermark(org.apache.flink.streaming.api.watermark.Watermark) ResultPartitionWriter(org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter) CompletableFuture(java.util.concurrent.CompletableFuture) AtomicReference(java.util.concurrent.atomic.AtomicReference) ArrayList(java.util.ArrayList) OperatorSnapshotFutures(org.apache.flink.streaming.api.operators.OperatorSnapshotFutures) StreamRecord(org.apache.flink.streaming.runtime.streamrecord.StreamRecord) StreamTaskStateInitializer(org.apache.flink.streaming.api.operators.StreamTaskStateInitializer) OperatorMetricGroup(org.apache.flink.metrics.groups.OperatorMetricGroup) NoOpStreamTask(org.apache.flink.streaming.runtime.tasks.StreamTaskTest.NoOpStreamTask) ChannelStateWriter(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriter) StreamMap(org.apache.flink.streaming.api.operators.StreamMap) MockEnvironment(org.apache.flink.runtime.operators.testutils.MockEnvironment) CheckpointStorageLocationReference(org.apache.flink.runtime.state.CheckpointStorageLocationReference) RunnableFuture(java.util.concurrent.RunnableFuture) SnapshotResult(org.apache.flink.runtime.state.SnapshotResult) ChannelStateWriterImpl(org.apache.flink.runtime.checkpoint.channel.ChannelStateWriterImpl) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) IOException(java.io.IOException) AbstractStreamOperator(org.apache.flink.streaming.api.operators.AbstractStreamOperator) ExecutionException(java.util.concurrent.ExecutionException) TimeUnit(java.util.concurrent.TimeUnit) DoneFuture(org.apache.flink.runtime.state.DoneFuture) LatencyMarker(org.apache.flink.streaming.runtime.streamrecord.LatencyMarker) CheckpointStreamFactory(org.apache.flink.runtime.state.CheckpointStreamFactory) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) ResultSubpartitionInfo(org.apache.flink.runtime.checkpoint.channel.ResultSubpartitionInfo) CheckpointMetricsBuilder(org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder) CheckpointMetaData(org.apache.flink.runtime.checkpoint.CheckpointMetaData) AbstractStreamOperator(org.apache.flink.streaming.api.operators.AbstractStreamOperator) TestTaskStateManager(org.apache.flink.runtime.state.TestTaskStateManager) StreamTaskStateInitializerImpl(org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl) MockEnvironment(org.apache.flink.runtime.operators.testutils.MockEnvironment) StreamMap(org.apache.flink.streaming.api.operators.StreamMap) Test(org.junit.Test)

Aggregations

IOException (java.io.IOException)2 ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1 CompletableFuture (java.util.concurrent.CompletableFuture)1 CountDownLatch (java.util.concurrent.CountDownLatch)1 ExecutionException (java.util.concurrent.ExecutionException)1 Executors (java.util.concurrent.Executors)1 RunnableFuture (java.util.concurrent.RunnableFuture)1 TimeUnit (java.util.concurrent.TimeUnit)1 AtomicReference (java.util.concurrent.atomic.AtomicReference)1 MapFunction (org.apache.flink.api.common.functions.MapFunction)1 BasicTypeInfo (org.apache.flink.api.common.typeinfo.BasicTypeInfo)1 StringSerializer (org.apache.flink.api.common.typeutils.base.StringSerializer)1 SavepointFormatType (org.apache.flink.core.execution.SavepointFormatType)1 CloseableRegistry (org.apache.flink.core.fs.CloseableRegistry)1 OneShotLatch (org.apache.flink.core.testutils.OneShotLatch)1 OperatorMetricGroup (org.apache.flink.metrics.groups.OperatorMetricGroup)1 CheckpointException (org.apache.flink.runtime.checkpoint.CheckpointException)1 CheckpointMetaData (org.apache.flink.runtime.checkpoint.CheckpointMetaData)1 CheckpointMetricsBuilder (org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder)1