Search in sources :

Example 6 with ExternalizedCheckpointSettings

use of org.apache.flink.runtime.jobgraph.tasks.ExternalizedCheckpointSettings in project flink by apache.

the class StreamingJobGraphGenerator method configureCheckpointing.

private void configureCheckpointing() {
    CheckpointConfig cfg = streamGraph.getCheckpointConfig();
    long interval = cfg.getCheckpointInterval();
    if (interval > 0) {
        // check if a restart strategy has been set, if not then set the FixedDelayRestartStrategy
        if (streamGraph.getExecutionConfig().getRestartStrategy() == null) {
            // if the user enabled checkpointing, the default number of exec retries is infinite.
            streamGraph.getExecutionConfig().setRestartStrategy(RestartStrategies.fixedDelayRestart(Integer.MAX_VALUE, DEFAULT_RESTART_DELAY));
        }
    } else {
        // interval of max value means disable periodic checkpoint
        interval = Long.MAX_VALUE;
    }
    // collect the vertices that receive "trigger checkpoint" messages.
    // currently, these are all the sources
    List<JobVertexID> triggerVertices = new ArrayList<>();
    // collect the vertices that need to acknowledge the checkpoint
    // currently, these are all vertices
    List<JobVertexID> ackVertices = new ArrayList<>(jobVertices.size());
    // collect the vertices that receive "commit checkpoint" messages
    // currently, these are all vertices
    List<JobVertexID> commitVertices = new ArrayList<>();
    for (JobVertex vertex : jobVertices.values()) {
        if (vertex.isInputVertex()) {
            triggerVertices.add(vertex.getID());
        }
        commitVertices.add(vertex.getID());
        ackVertices.add(vertex.getID());
    }
    ExternalizedCheckpointSettings externalizedCheckpointSettings;
    if (cfg.isExternalizedCheckpointsEnabled()) {
        CheckpointConfig.ExternalizedCheckpointCleanup cleanup = cfg.getExternalizedCheckpointCleanup();
        // Sanity check
        if (cleanup == null) {
            throw new IllegalStateException("Externalized checkpoints enabled, but no cleanup mode configured.");
        }
        externalizedCheckpointSettings = ExternalizedCheckpointSettings.externalizeCheckpoints(cleanup.deleteOnCancellation());
    } else {
        externalizedCheckpointSettings = ExternalizedCheckpointSettings.none();
    }
    CheckpointingMode mode = cfg.getCheckpointingMode();
    boolean isExactlyOnce;
    if (mode == CheckpointingMode.EXACTLY_ONCE) {
        isExactlyOnce = true;
    } else if (mode == CheckpointingMode.AT_LEAST_ONCE) {
        isExactlyOnce = false;
    } else {
        throw new IllegalStateException("Unexpected checkpointing mode. " + "Did not expect there to be another checkpointing mode besides " + "exactly-once or at-least-once.");
    }
    JobSnapshottingSettings settings = new JobSnapshottingSettings(triggerVertices, ackVertices, commitVertices, interval, cfg.getCheckpointTimeout(), cfg.getMinPauseBetweenCheckpoints(), cfg.getMaxConcurrentCheckpoints(), externalizedCheckpointSettings, streamGraph.getStateBackend(), isExactlyOnce);
    jobGraph.setSnapshotSettings(settings);
}
Also used : JobVertex(org.apache.flink.runtime.jobgraph.JobVertex) CheckpointConfig(org.apache.flink.streaming.api.environment.CheckpointConfig) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) ExternalizedCheckpointSettings(org.apache.flink.runtime.jobgraph.tasks.ExternalizedCheckpointSettings) ArrayList(java.util.ArrayList) CheckpointingMode(org.apache.flink.streaming.api.CheckpointingMode) JobSnapshottingSettings(org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings)

Aggregations

ExternalizedCheckpointSettings (org.apache.flink.runtime.jobgraph.tasks.ExternalizedCheckpointSettings)6 JobSnapshottingSettings (org.apache.flink.runtime.jobgraph.tasks.JobSnapshottingSettings)4 AccessExecutionGraph (org.apache.flink.runtime.executiongraph.AccessExecutionGraph)3 Test (org.junit.Test)3 JsonNode (com.fasterxml.jackson.databind.JsonNode)2 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)2 JobID (org.apache.flink.api.common.JobID)2 JobVertexID (org.apache.flink.runtime.jobgraph.JobVertexID)2 JsonGenerator (com.fasterxml.jackson.core.JsonGenerator)1 File (java.io.File)1 StringWriter (java.io.StringWriter)1 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 ExecutionAttemptID (org.apache.flink.runtime.executiongraph.ExecutionAttemptID)1 ExecutionJobVertex (org.apache.flink.runtime.executiongraph.ExecutionJobVertex)1 ExecutionVertex (org.apache.flink.runtime.executiongraph.ExecutionVertex)1 JobVertex (org.apache.flink.runtime.jobgraph.JobVertex)1 AcknowledgeCheckpoint (org.apache.flink.runtime.messages.checkpoint.AcknowledgeCheckpoint)1 ExecutionGraphHolder (org.apache.flink.runtime.webmonitor.ExecutionGraphHolder)1 ArchivedJson (org.apache.flink.runtime.webmonitor.history.ArchivedJson)1