Search in sources :

Example 1 with TestSchedulingStrategy

use of org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy in project flink by apache.

the class DefaultSchedulerTest method vertexIsResetBeforeRestarted.

@Test
public void vertexIsResetBeforeRestarted() throws Exception {
    final JobGraph jobGraph = singleNonParallelJobVertexJobGraph();
    final TestSchedulingStrategy.Factory schedulingStrategyFactory = new TestSchedulingStrategy.Factory();
    final DefaultScheduler scheduler = createScheduler(jobGraph, ComponentMainThreadExecutorServiceAdapter.forMainThread(), schedulingStrategyFactory);
    final TestSchedulingStrategy schedulingStrategy = schedulingStrategyFactory.getLastCreatedSchedulingStrategy();
    final SchedulingTopology topology = schedulingStrategy.getSchedulingTopology();
    scheduler.startScheduling();
    final SchedulingExecutionVertex onlySchedulingVertex = Iterables.getOnlyElement(topology.getVertices());
    schedulingStrategy.schedule(Collections.singletonList(onlySchedulingVertex.getId()));
    final ArchivedExecutionVertex onlyExecutionVertex = Iterables.getOnlyElement(scheduler.requestJob().getArchivedExecutionGraph().getAllExecutionVertices());
    final ExecutionAttemptID attemptId = onlyExecutionVertex.getCurrentExecutionAttempt().getAttemptId();
    scheduler.updateTaskExecutionState(createFailedTaskExecutionState(attemptId));
    taskRestartExecutor.triggerScheduledTasks();
    assertThat(schedulingStrategy.getReceivedVerticesToRestart(), hasSize(1));
    assertThat(onlySchedulingVertex.getState(), is(equalTo(ExecutionState.CREATED)));
}
Also used : JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) SchedulingExecutionVertex(org.apache.flink.runtime.scheduler.strategy.SchedulingExecutionVertex) ExecutionAttemptID(org.apache.flink.runtime.executiongraph.ExecutionAttemptID) ArchivedExecutionVertex(org.apache.flink.runtime.executiongraph.ArchivedExecutionVertex) TestingCheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.TestingCheckpointRecoveryFactory) SchedulingStrategyFactory(org.apache.flink.runtime.scheduler.strategy.SchedulingStrategyFactory) TestFailoverStrategyFactory(org.apache.flink.runtime.executiongraph.utils.TestFailoverStrategyFactory) CheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory) SchedulingTopology(org.apache.flink.runtime.scheduler.strategy.SchedulingTopology) TestSchedulingStrategy(org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy) AdaptiveSchedulerTest(org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerTest) Test(org.junit.Test)

Example 2 with TestSchedulingStrategy

use of org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy in project flink by apache.

the class DefaultSchedulerTest method deployTasksOnlyWhenAllSlotRequestsAreFulfilled.

@Test
public void deployTasksOnlyWhenAllSlotRequestsAreFulfilled() throws Exception {
    final JobGraph jobGraph = singleJobVertexJobGraph(4);
    final JobVertexID onlyJobVertexId = getOnlyJobVertex(jobGraph).getID();
    testExecutionSlotAllocator.disableAutoCompletePendingRequests();
    final TestSchedulingStrategy.Factory schedulingStrategyFactory = new TestSchedulingStrategy.Factory();
    final DefaultScheduler scheduler = createScheduler(jobGraph, ComponentMainThreadExecutorServiceAdapter.forMainThread(), schedulingStrategyFactory);
    final TestSchedulingStrategy schedulingStrategy = schedulingStrategyFactory.getLastCreatedSchedulingStrategy();
    scheduler.startScheduling();
    final List<ExecutionVertexID> verticesToSchedule = Arrays.asList(new ExecutionVertexID(onlyJobVertexId, 0), new ExecutionVertexID(onlyJobVertexId, 1), new ExecutionVertexID(onlyJobVertexId, 2), new ExecutionVertexID(onlyJobVertexId, 3));
    schedulingStrategy.schedule(verticesToSchedule);
    assertThat(testExecutionVertexOperations.getDeployedVertices(), hasSize(0));
    testExecutionSlotAllocator.completePendingRequest(verticesToSchedule.get(0));
    assertThat(testExecutionVertexOperations.getDeployedVertices(), hasSize(0));
    testExecutionSlotAllocator.completePendingRequests();
    assertThat(testExecutionVertexOperations.getDeployedVertices(), hasSize(4));
}
Also used : JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) ExecutionVertexID(org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) TestingCheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.TestingCheckpointRecoveryFactory) SchedulingStrategyFactory(org.apache.flink.runtime.scheduler.strategy.SchedulingStrategyFactory) TestFailoverStrategyFactory(org.apache.flink.runtime.executiongraph.utils.TestFailoverStrategyFactory) CheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory) TestSchedulingStrategy(org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy) AdaptiveSchedulerTest(org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerTest) Test(org.junit.Test)

Example 3 with TestSchedulingStrategy

use of org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy in project flink by apache.

the class DefaultSchedulerTest method scheduledVertexOrderFromSchedulingStrategyIsRespected.

@Test
public void scheduledVertexOrderFromSchedulingStrategyIsRespected() throws Exception {
    final JobGraph jobGraph = singleJobVertexJobGraph(10);
    final JobVertexID onlyJobVertexId = getOnlyJobVertex(jobGraph).getID();
    final List<ExecutionVertexID> desiredScheduleOrder = Arrays.asList(new ExecutionVertexID(onlyJobVertexId, 4), new ExecutionVertexID(onlyJobVertexId, 0), new ExecutionVertexID(onlyJobVertexId, 3), new ExecutionVertexID(onlyJobVertexId, 1), new ExecutionVertexID(onlyJobVertexId, 2));
    final TestSchedulingStrategy.Factory schedulingStrategyFactory = new TestSchedulingStrategy.Factory();
    createScheduler(jobGraph, ComponentMainThreadExecutorServiceAdapter.forMainThread(), schedulingStrategyFactory);
    final TestSchedulingStrategy schedulingStrategy = schedulingStrategyFactory.getLastCreatedSchedulingStrategy();
    schedulingStrategy.schedule(desiredScheduleOrder);
    final List<ExecutionVertexID> deployedExecutionVertices = testExecutionVertexOperations.getDeployedVertices();
    assertEquals(desiredScheduleOrder, deployedExecutionVertices);
}
Also used : JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) ExecutionVertexID(org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) TestingCheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.TestingCheckpointRecoveryFactory) SchedulingStrategyFactory(org.apache.flink.runtime.scheduler.strategy.SchedulingStrategyFactory) TestFailoverStrategyFactory(org.apache.flink.runtime.executiongraph.utils.TestFailoverStrategyFactory) CheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory) TestSchedulingStrategy(org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy) AdaptiveSchedulerTest(org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerTest) Test(org.junit.Test)

Example 4 with TestSchedulingStrategy

use of org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy in project flink by apache.

the class DefaultSchedulerTest method testRestartVerticesOnFailuresInScheduling.

private void testRestartVerticesOnFailuresInScheduling(Consumer<ExecutionVertexID> actionsToTriggerTaskFailure) throws Exception {
    final int parallelism = 2;
    final JobVertex v1 = createVertex("vertex1", parallelism);
    final JobVertex v2 = createVertex("vertex2", parallelism);
    v2.connectNewDataSetAsInput(v1, DistributionPattern.POINTWISE, ResultPartitionType.PIPELINED);
    final JobGraph jobGraph = JobGraphTestUtils.streamingJobGraph(v1, v2);
    testExecutionSlotAllocator.disableAutoCompletePendingRequests();
    final TestSchedulingStrategy.Factory schedulingStrategyFactory = new TestSchedulingStrategy.Factory();
    final DefaultScheduler scheduler = createScheduler(jobGraph, ComponentMainThreadExecutorServiceAdapter.forMainThread(), schedulingStrategyFactory, new RestartPipelinedRegionFailoverStrategy.Factory());
    final TestSchedulingStrategy schedulingStrategy = schedulingStrategyFactory.getLastCreatedSchedulingStrategy();
    scheduler.startScheduling();
    final ExecutionVertexID vid11 = new ExecutionVertexID(v1.getID(), 0);
    final ExecutionVertexID vid12 = new ExecutionVertexID(v1.getID(), 1);
    final ExecutionVertexID vid21 = new ExecutionVertexID(v2.getID(), 0);
    final ExecutionVertexID vid22 = new ExecutionVertexID(v2.getID(), 1);
    schedulingStrategy.schedule(Arrays.asList(vid11, vid12, vid21, vid22));
    assertThat(testExecutionSlotAllocator.getPendingRequests().keySet(), hasSize(4));
    actionsToTriggerTaskFailure.accept(vid11);
    final Iterator<ArchivedExecutionVertex> vertexIterator = scheduler.requestJob().getArchivedExecutionGraph().getAllExecutionVertices().iterator();
    final ArchivedExecutionVertex ev11 = vertexIterator.next();
    final ArchivedExecutionVertex ev12 = vertexIterator.next();
    final ArchivedExecutionVertex ev21 = vertexIterator.next();
    final ArchivedExecutionVertex ev22 = vertexIterator.next();
    // ev11 and ev21 needs to be restarted because it is pipelined region failover and
    // they are in the same region. ev12 and ev22 will not be affected
    assertThat(testExecutionSlotAllocator.getPendingRequests().keySet(), hasSize(2));
    assertThat(ev11.getExecutionState(), is(ExecutionState.FAILED));
    assertThat(ev21.getExecutionState(), is(ExecutionState.CANCELED));
    assertThat(ev12.getExecutionState(), is(ExecutionState.SCHEDULED));
    assertThat(ev22.getExecutionState(), is(ExecutionState.SCHEDULED));
    taskRestartExecutor.triggerScheduledTasks();
    assertThat(schedulingStrategy.getReceivedVerticesToRestart(), containsInAnyOrder(vid11, vid21));
}
Also used : JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) JobVertex(org.apache.flink.runtime.jobgraph.JobVertex) ExecutionVertexID(org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID) ArchivedExecutionVertex(org.apache.flink.runtime.executiongraph.ArchivedExecutionVertex) TestingCheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.TestingCheckpointRecoveryFactory) SchedulingStrategyFactory(org.apache.flink.runtime.scheduler.strategy.SchedulingStrategyFactory) TestFailoverStrategyFactory(org.apache.flink.runtime.executiongraph.utils.TestFailoverStrategyFactory) CheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory) SchedulerTestingUtils.acknowledgePendingCheckpoint(org.apache.flink.runtime.scheduler.SchedulerTestingUtils.acknowledgePendingCheckpoint) TestSchedulingStrategy(org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy) RestartPipelinedRegionFailoverStrategy(org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy)

Example 5 with TestSchedulingStrategy

use of org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy in project flink by apache.

the class DefaultSchedulerTest method scheduleOnlyIfVertexIsCreated.

@Test
public void scheduleOnlyIfVertexIsCreated() throws Exception {
    final JobGraph jobGraph = singleNonParallelJobVertexJobGraph();
    final TestSchedulingStrategy.Factory schedulingStrategyFactory = new TestSchedulingStrategy.Factory();
    final DefaultScheduler scheduler = createScheduler(jobGraph, ComponentMainThreadExecutorServiceAdapter.forMainThread(), schedulingStrategyFactory);
    final TestSchedulingStrategy schedulingStrategy = schedulingStrategyFactory.getLastCreatedSchedulingStrategy();
    final SchedulingTopology topology = schedulingStrategy.getSchedulingTopology();
    scheduler.startScheduling();
    final ExecutionVertexID onlySchedulingVertexId = Iterables.getOnlyElement(topology.getVertices()).getId();
    // Schedule the vertex to get it to a non-CREATED state
    schedulingStrategy.schedule(Collections.singletonList(onlySchedulingVertexId));
    // The scheduling of a non-CREATED vertex will result in IllegalStateException
    try {
        schedulingStrategy.schedule(Collections.singletonList(onlySchedulingVertexId));
        fail("IllegalStateException should happen");
    } catch (IllegalStateException e) {
    // expected exception
    }
}
Also used : JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) ExecutionVertexID(org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID) TestingCheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.TestingCheckpointRecoveryFactory) SchedulingStrategyFactory(org.apache.flink.runtime.scheduler.strategy.SchedulingStrategyFactory) TestFailoverStrategyFactory(org.apache.flink.runtime.executiongraph.utils.TestFailoverStrategyFactory) CheckpointRecoveryFactory(org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory) SchedulingTopology(org.apache.flink.runtime.scheduler.strategy.SchedulingTopology) TestSchedulingStrategy(org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy) AdaptiveSchedulerTest(org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerTest) Test(org.junit.Test)

Aggregations

CheckpointRecoveryFactory (org.apache.flink.runtime.checkpoint.CheckpointRecoveryFactory)5 TestingCheckpointRecoveryFactory (org.apache.flink.runtime.checkpoint.TestingCheckpointRecoveryFactory)5 TestFailoverStrategyFactory (org.apache.flink.runtime.executiongraph.utils.TestFailoverStrategyFactory)5 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)5 SchedulingStrategyFactory (org.apache.flink.runtime.scheduler.strategy.SchedulingStrategyFactory)5 TestSchedulingStrategy (org.apache.flink.runtime.scheduler.strategy.TestSchedulingStrategy)5 AdaptiveSchedulerTest (org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerTest)4 ExecutionVertexID (org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID)4 Test (org.junit.Test)4 ArchivedExecutionVertex (org.apache.flink.runtime.executiongraph.ArchivedExecutionVertex)2 JobVertexID (org.apache.flink.runtime.jobgraph.JobVertexID)2 SchedulingTopology (org.apache.flink.runtime.scheduler.strategy.SchedulingTopology)2 ExecutionAttemptID (org.apache.flink.runtime.executiongraph.ExecutionAttemptID)1 RestartPipelinedRegionFailoverStrategy (org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy)1 JobVertex (org.apache.flink.runtime.jobgraph.JobVertex)1 SchedulerTestingUtils.acknowledgePendingCheckpoint (org.apache.flink.runtime.scheduler.SchedulerTestingUtils.acknowledgePendingCheckpoint)1 SchedulingExecutionVertex (org.apache.flink.runtime.scheduler.strategy.SchedulingExecutionVertex)1