Examples with VertexParallelism org.apache.flink.runtime.scheduler.adaptive.allocator.VertexParallelism used on opensource projects

Example 1 with VertexParallelism

use of org.apache.flink.runtime.scheduler.adaptive.allocator.VertexParallelism in project flink by apache.

the class AdaptiveScheduler method tryToAssignSlots.

@Override
public CreatingExecutionGraph.AssignmentResult tryToAssignSlots(CreatingExecutionGraph.ExecutionGraphWithVertexParallelism executionGraphWithVertexParallelism) {
    final ExecutionGraph executionGraph = executionGraphWithVertexParallelism.getExecutionGraph();
    executionGraph.start(componentMainThreadExecutor);
    executionGraph.transitionToRunning();
    executionGraph.setInternalTaskFailuresListener(new UpdateSchedulerNgOnInternalFailuresListener(this));
    final VertexParallelism vertexParallelism = executionGraphWithVertexParallelism.getVertexParallelism();
    return slotAllocator.tryReserveResources(vertexParallelism).map(reservedSlots -> CreatingExecutionGraph.AssignmentResult.success(assignSlotsToExecutionGraph(executionGraph, reservedSlots))).orElseGet(CreatingExecutionGraph.AssignmentResult::notPossible);
}

Example 2 with VertexParallelism

use of org.apache.flink.runtime.scheduler.adaptive.allocator.VertexParallelism in project flink by apache.

the class AdaptiveScheduler method createExecutionGraphWithAvailableResourcesAsync.

private CompletableFuture<CreatingExecutionGraph.ExecutionGraphWithVertexParallelism> createExecutionGraphWithAvailableResourcesAsync() {
    final VertexParallelism vertexParallelism;
    final VertexParallelismStore adjustedParallelismStore;
    try {
        vertexParallelism = determineParallelism(slotAllocator);
        JobGraph adjustedJobGraph = jobInformation.copyJobGraph();
        for (JobVertex vertex : adjustedJobGraph.getVertices()) {
            JobVertexID id = vertex.getID();
            // use the determined "available parallelism" to use
            // the resources we have access to
            vertex.setParallelism(vertexParallelism.getParallelism(id));
        }
        // use the originally configured max parallelism
        // as the default for consistent runs
        adjustedParallelismStore = computeVertexParallelismStoreForExecution(adjustedJobGraph, executionMode, (vertex) -> {
            VertexParallelismInformation vertexParallelismInfo = initialParallelismStore.getParallelismInfo(vertex.getID());
            return vertexParallelismInfo.getMaxParallelism();
        });
    } catch (Exception exception) {
        return FutureUtils.completedExceptionally(exception);
    }
    return createExecutionGraphAndRestoreStateAsync(adjustedParallelismStore).thenApply(executionGraph -> CreatingExecutionGraph.ExecutionGraphWithVertexParallelism.create(executionGraph, vertexParallelism));
}