Search in sources :

Example 6 with TestingPhysicalSlot

use of org.apache.flink.runtime.scheduler.TestingPhysicalSlot in project flink by apache.

the class SharedSlotTest method testReturnLogicalSlotRejectsAliveSlots.

@Test(expected = IllegalStateException.class)
public void testReturnLogicalSlotRejectsAliveSlots() {
    final TestingPhysicalSlot physicalSlot = TestingPhysicalSlot.builder().build();
    final SharedSlot sharedSlot = new SharedSlot(new SlotRequestId(), physicalSlot, false, () -> {
    });
    final LogicalSlot logicalSlot = sharedSlot.allocateLogicalSlot();
    sharedSlot.returnLogicalSlot(logicalSlot);
}
Also used : SlotRequestId(org.apache.flink.runtime.jobmaster.SlotRequestId) TestingPhysicalSlot(org.apache.flink.runtime.scheduler.TestingPhysicalSlot) LogicalSlot(org.apache.flink.runtime.jobmaster.LogicalSlot) Test(org.junit.Test)

Example 7 with TestingPhysicalSlot

use of org.apache.flink.runtime.scheduler.TestingPhysicalSlot in project flink by apache.

the class SharedSlotTest method testCanReturnLogicalSlotDuringRelease.

@Test
public void testCanReturnLogicalSlotDuringRelease() {
    final TestingPhysicalSlot physicalSlot = TestingPhysicalSlot.builder().build();
    final SharedSlot sharedSlot = new SharedSlot(new SlotRequestId(), physicalSlot, false, () -> {
    });
    final LogicalSlot logicalSlot1 = sharedSlot.allocateLogicalSlot();
    final LogicalSlot logicalSlot2 = sharedSlot.allocateLogicalSlot();
    // both slots try to release the other one, simulating that the failure of one execution due
    // to the release also fails others
    logicalSlot1.tryAssignPayload(new TestLogicalSlotPayload(cause -> {
        if (logicalSlot2.isAlive()) {
            logicalSlot2.releaseSlot(cause);
        }
    }));
    logicalSlot2.tryAssignPayload(new TestLogicalSlotPayload(cause -> {
        if (logicalSlot1.isAlive()) {
            logicalSlot1.releaseSlot(cause);
        }
    }));
    sharedSlot.release(new Exception("test"));
    // if all logical slots were released, and the sharedSlot no longer allows the allocation of
    // logical slots, then the slot release was completed
    assertThat(logicalSlot1.isAlive(), is(false));
    assertThat(logicalSlot2.isAlive(), is(false));
    try {
        sharedSlot.allocateLogicalSlot();
        fail("Allocation of logical slot should have failed because the slot was released.");
    } catch (IllegalStateException expected) {
    }
}
Also used : SlotRequestId(org.apache.flink.runtime.jobmaster.SlotRequestId) CoreMatchers.is(org.hamcrest.CoreMatchers.is) TestingPhysicalSlotPayload(org.apache.flink.runtime.jobmaster.slotpool.TestingPhysicalSlotPayload) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) CoreMatchers.not(org.hamcrest.CoreMatchers.not) Locality(org.apache.flink.runtime.jobmanager.scheduler.Locality) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) LogicalSlot(org.apache.flink.runtime.jobmaster.LogicalSlot) Test(org.junit.Test) CompletableFuture(java.util.concurrent.CompletableFuture) TestingLogicalSlotBuilder(org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder) Consumer(java.util.function.Consumer) Assert.assertThat(org.junit.Assert.assertThat) TestingPhysicalSlot(org.apache.flink.runtime.scheduler.TestingPhysicalSlot) TestLogger(org.apache.flink.util.TestLogger) Assert.fail(org.junit.Assert.fail) SlotRequestId(org.apache.flink.runtime.jobmaster.SlotRequestId) CoreMatchers.nullValue(org.hamcrest.CoreMatchers.nullValue) TestingPhysicalSlot(org.apache.flink.runtime.scheduler.TestingPhysicalSlot) LogicalSlot(org.apache.flink.runtime.jobmaster.LogicalSlot) Test(org.junit.Test)

Example 8 with TestingPhysicalSlot

use of org.apache.flink.runtime.scheduler.TestingPhysicalSlot in project flink by apache.

the class PreferredAllocationRequestSlotMatchingStrategyTest method testNewSlotsAreMatchedAgainstPreferredAllocationIDs.

/**
 * This test ensures that new slots are matched against the preferred allocationIds of the
 * pending requests.
 */
@Test
public void testNewSlotsAreMatchedAgainstPreferredAllocationIDs() throws Exception {
    final PreferredAllocationRequestSlotMatchingStrategy strategy = PreferredAllocationRequestSlotMatchingStrategy.INSTANCE;
    final AllocationID allocationId1 = new AllocationID();
    final AllocationID allocationId2 = new AllocationID();
    final Collection<TestingPhysicalSlot> slots = Arrays.asList(TestingPhysicalSlot.builder().withAllocationID(allocationId1).build(), TestingPhysicalSlot.builder().withAllocationID(allocationId2).build());
    final Collection<PendingRequest> pendingRequests = Arrays.asList(PendingRequest.createNormalRequest(new SlotRequestId(), ResourceProfile.UNKNOWN, Collections.singleton(allocationId2)), PendingRequest.createNormalRequest(new SlotRequestId(), ResourceProfile.UNKNOWN, Collections.singleton(allocationId1)));
    final Collection<RequestSlotMatchingStrategy.RequestSlotMatch> requestSlotMatches = strategy.matchRequestsAndSlots(slots, pendingRequests);
    assertThat(requestSlotMatches).hasSize(2);
    for (RequestSlotMatchingStrategy.RequestSlotMatch requestSlotMatch : requestSlotMatches) {
        assertThat(requestSlotMatch.getPendingRequest().getPreferredAllocations()).contains(requestSlotMatch.getSlot().getAllocationId());
    }
}
Also used : SlotRequestId(org.apache.flink.runtime.jobmaster.SlotRequestId) TestingPhysicalSlot(org.apache.flink.runtime.scheduler.TestingPhysicalSlot) AllocationID(org.apache.flink.runtime.clusterframework.types.AllocationID) Test(org.junit.jupiter.api.Test)

Example 9 with TestingPhysicalSlot

use of org.apache.flink.runtime.scheduler.TestingPhysicalSlot in project flink by apache.

the class ExecutionTest method testSlotReleaseAtomicallyReleasesExecution.

/**
 * Tests that a slot release will atomically release the assigned {@link Execution}.
 */
@Test
public void testSlotReleaseAtomicallyReleasesExecution() throws Exception {
    final JobVertex jobVertex = createNoOpJobVertex();
    final TestingPhysicalSlotProvider physicalSlotProvider = TestingPhysicalSlotProvider.createWithLimitedAmountOfPhysicalSlots(1);
    final SchedulerBase scheduler = SchedulerTestingUtils.newSchedulerBuilder(JobGraphTestUtils.streamingJobGraph(jobVertex), testMainThreadUtil.getMainThreadExecutor()).setExecutionSlotAllocatorFactory(SchedulerTestingUtils.newSlotSharingExecutionSlotAllocatorFactory(physicalSlotProvider)).build();
    final Execution execution = scheduler.getExecutionJobVertex(jobVertex.getID()).getTaskVertices()[0].getCurrentExecutionAttempt();
    testMainThreadUtil.execute(scheduler::startScheduling);
    // wait until the slot has been requested
    physicalSlotProvider.awaitAllSlotRequests();
    TestingPhysicalSlot physicalSlot = physicalSlotProvider.getFirstResponseOrFail().get();
    testMainThreadUtil.execute(() -> {
        assertThat(execution.getAssignedAllocationID(), is(physicalSlot.getAllocationId()));
        physicalSlot.releasePayload(new FlinkException("Test exception"));
        assertThat(execution.getReleaseFuture().isDone(), is(true));
    });
}
Also used : TestingPhysicalSlot(org.apache.flink.runtime.scheduler.TestingPhysicalSlot) JobVertex(org.apache.flink.runtime.jobgraph.JobVertex) TestingPhysicalSlotProvider(org.apache.flink.runtime.scheduler.TestingPhysicalSlotProvider) SchedulerBase(org.apache.flink.runtime.scheduler.SchedulerBase) FlinkException(org.apache.flink.util.FlinkException) Test(org.junit.Test)

Example 10 with TestingPhysicalSlot

use of org.apache.flink.runtime.scheduler.TestingPhysicalSlot in project flink by apache.

the class DefaultExecutionGraphDeploymentTest method testExecutionGraphIsDeployedInTopologicalOrder.

/**
 * Tests that the {@link ExecutionGraph} is deployed in topological order.
 */
@Test
public void testExecutionGraphIsDeployedInTopologicalOrder() throws Exception {
    final int sourceParallelism = 2;
    final int sinkParallelism = 1;
    final JobVertex sourceVertex = new JobVertex("source");
    sourceVertex.setInvokableClass(NoOpInvokable.class);
    sourceVertex.setParallelism(sourceParallelism);
    final JobVertex sinkVertex = new JobVertex("sink");
    sinkVertex.setInvokableClass(NoOpInvokable.class);
    sinkVertex.setParallelism(sinkParallelism);
    sinkVertex.connectNewDataSetAsInput(sourceVertex, DistributionPattern.POINTWISE, ResultPartitionType.PIPELINED);
    final int numberTasks = sourceParallelism + sinkParallelism;
    final ArrayBlockingQueue<ExecutionAttemptID> submittedTasksQueue = new ArrayBlockingQueue<>(numberTasks);
    TestingTaskExecutorGatewayBuilder testingTaskExecutorGatewayBuilder = new TestingTaskExecutorGatewayBuilder();
    testingTaskExecutorGatewayBuilder.setSubmitTaskConsumer((taskDeploymentDescriptor, jobMasterId) -> {
        submittedTasksQueue.offer(taskDeploymentDescriptor.getExecutionAttemptId());
        return CompletableFuture.completedFuture(Acknowledge.get());
    });
    final TaskManagerLocation taskManagerLocation = new LocalTaskManagerLocation();
    final TestingTaskExecutorGateway taskExecutorGateway = testingTaskExecutorGatewayBuilder.createTestingTaskExecutorGateway();
    final RpcTaskManagerGateway taskManagerGateway = new RpcTaskManagerGateway(taskExecutorGateway, JobMasterId.generate());
    final JobGraph jobGraph = JobGraphTestUtils.streamingJobGraph(sourceVertex, sinkVertex);
    final TestingPhysicalSlotProvider physicalSlotProvider = TestingPhysicalSlotProvider.createWithoutImmediatePhysicalSlotCreation();
    final SchedulerBase scheduler = SchedulerTestingUtils.newSchedulerBuilder(jobGraph, ComponentMainThreadExecutorServiceAdapter.forMainThread()).setExecutionSlotAllocatorFactory(SchedulerTestingUtils.newSlotSharingExecutionSlotAllocatorFactory(physicalSlotProvider)).setFutureExecutor(new DirectScheduledExecutorService()).build();
    final ExecutionGraph executionGraph = scheduler.getExecutionGraph();
    scheduler.startScheduling();
    // change the order in which the futures are completed
    final List<CompletableFuture<TestingPhysicalSlot>> shuffledFutures = new ArrayList<>(physicalSlotProvider.getResponses().values());
    Collections.shuffle(shuffledFutures);
    for (CompletableFuture<TestingPhysicalSlot> slotFuture : shuffledFutures) {
        slotFuture.complete(TestingPhysicalSlot.builder().withTaskManagerLocation(taskManagerLocation).withTaskManagerGateway(taskManagerGateway).build());
    }
    final List<ExecutionAttemptID> submittedTasks = new ArrayList<>(numberTasks);
    for (int i = 0; i < numberTasks; i++) {
        submittedTasks.add(submittedTasksQueue.take());
    }
    final Collection<ExecutionAttemptID> firstStage = new ArrayList<>(sourceParallelism);
    for (ExecutionVertex taskVertex : executionGraph.getJobVertex(sourceVertex.getID()).getTaskVertices()) {
        firstStage.add(taskVertex.getCurrentExecutionAttempt().getAttemptId());
    }
    final Collection<ExecutionAttemptID> secondStage = new ArrayList<>(sinkParallelism);
    for (ExecutionVertex taskVertex : executionGraph.getJobVertex(sinkVertex.getID()).getTaskVertices()) {
        secondStage.add(taskVertex.getCurrentExecutionAttempt().getAttemptId());
    }
    assertThat(submittedTasks, new ExecutionStageMatcher(Arrays.asList(firstStage, secondStage)));
}
Also used : TestingPhysicalSlot(org.apache.flink.runtime.scheduler.TestingPhysicalSlot) RpcTaskManagerGateway(org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway) TaskManagerLocation(org.apache.flink.runtime.taskmanager.TaskManagerLocation) LocalTaskManagerLocation(org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation) DirectScheduledExecutorService(org.apache.flink.runtime.testutils.DirectScheduledExecutorService) ArrayList(java.util.ArrayList) TestingTaskExecutorGatewayBuilder(org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGatewayBuilder) JobGraph(org.apache.flink.runtime.jobgraph.JobGraph) CompletableFuture(java.util.concurrent.CompletableFuture) JobVertex(org.apache.flink.runtime.jobgraph.JobVertex) ArrayBlockingQueue(java.util.concurrent.ArrayBlockingQueue) LocalTaskManagerLocation(org.apache.flink.runtime.taskmanager.LocalTaskManagerLocation) TestingPhysicalSlotProvider(org.apache.flink.runtime.scheduler.TestingPhysicalSlotProvider) SchedulerBase(org.apache.flink.runtime.scheduler.SchedulerBase) TestingTaskExecutorGateway(org.apache.flink.runtime.taskexecutor.TestingTaskExecutorGateway) Test(org.junit.Test)

Aggregations

TestingPhysicalSlot (org.apache.flink.runtime.scheduler.TestingPhysicalSlot)16 Test (org.junit.Test)15 SlotRequestId (org.apache.flink.runtime.jobmaster.SlotRequestId)13 LogicalSlot (org.apache.flink.runtime.jobmaster.LogicalSlot)8 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)4 CompletableFuture (java.util.concurrent.CompletableFuture)3 JobVertex (org.apache.flink.runtime.jobgraph.JobVertex)3 TestingLogicalSlotBuilder (org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder)3 TestingPhysicalSlotPayload (org.apache.flink.runtime.jobmaster.slotpool.TestingPhysicalSlotPayload)3 SchedulerBase (org.apache.flink.runtime.scheduler.SchedulerBase)3 TestingPhysicalSlotProvider (org.apache.flink.runtime.scheduler.TestingPhysicalSlotProvider)3 Consumer (java.util.function.Consumer)2 AllocationID (org.apache.flink.runtime.clusterframework.types.AllocationID)2 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)2 Locality (org.apache.flink.runtime.jobmanager.scheduler.Locality)2 TaskManagerLocation (org.apache.flink.runtime.taskmanager.TaskManagerLocation)2 TestLogger (org.apache.flink.util.TestLogger)2 CoreMatchers.equalTo (org.hamcrest.CoreMatchers.equalTo)2 CoreMatchers.is (org.hamcrest.CoreMatchers.is)2 CoreMatchers.not (org.hamcrest.CoreMatchers.not)2