Search in sources :

Example 1 with TestingClassLoaderLease

use of org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease in project flink by apache.

the class TaskAsyncCallTest method createTask.

private Task createTask(Class<? extends AbstractInvokable> invokableClass) throws Exception {
    final TestingClassLoaderLease classLoaderHandle = TestingClassLoaderLease.newBuilder().setGetOrResolveClassLoaderFunction((permanentBlobKeys, urls) -> TestingUserCodeClassLoader.newBuilder().setClassLoader(new TestUserCodeClassLoader()).build()).build();
    ResultPartitionConsumableNotifier consumableNotifier = new NoOpResultPartitionConsumableNotifier();
    PartitionProducerStateChecker partitionProducerStateChecker = mock(PartitionProducerStateChecker.class);
    Executor executor = mock(Executor.class);
    TaskMetricGroup taskMetricGroup = UnregisteredMetricGroups.createUnregisteredTaskMetricGroup();
    JobInformation jobInformation = new JobInformation(new JobID(), "Job Name", new SerializedValue<>(new ExecutionConfig()), new Configuration(), Collections.emptyList(), Collections.emptyList());
    TaskInformation taskInformation = new TaskInformation(new JobVertexID(), "Test Task", 1, 1, invokableClass.getName(), new Configuration());
    return new Task(jobInformation, taskInformation, new ExecutionAttemptID(), new AllocationID(), 0, 0, Collections.<ResultPartitionDeploymentDescriptor>emptyList(), Collections.<InputGateDeploymentDescriptor>emptyList(), mock(MemoryManager.class), mock(IOManager.class), shuffleEnvironment, new KvStateService(new KvStateRegistry(), null, null), mock(BroadcastVariableManager.class), new TaskEventDispatcher(), ExternalResourceInfoProvider.NO_EXTERNAL_RESOURCES, new TestTaskStateManager(), mock(TaskManagerActions.class), mock(InputSplitProvider.class), mock(CheckpointResponder.class), new NoOpTaskOperatorEventGateway(), new TestGlobalAggregateManager(), classLoaderHandle, mock(FileCache.class), new TestingTaskManagerRuntimeInfo(), taskMetricGroup, consumableNotifier, partitionProducerStateChecker, executor);
}
Also used : CheckpointMetricsBuilder(org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder) ShuffleEnvironment(org.apache.flink.runtime.shuffle.ShuffleEnvironment) IOManager(org.apache.flink.runtime.io.disk.iomanager.IOManager) InputSplitProvider(org.apache.flink.runtime.jobgraph.tasks.InputSplitProvider) Assert.assertThat(org.junit.Assert.assertThat) TestTaskStateManager(org.apache.flink.runtime.state.TestTaskStateManager) Future(java.util.concurrent.Future) CheckpointException(org.apache.flink.runtime.checkpoint.CheckpointException) JobInformation(org.apache.flink.runtime.executiongraph.JobInformation) KvStateService(org.apache.flink.runtime.taskexecutor.KvStateService) After(org.junit.After) TestGlobalAggregateManager(org.apache.flink.runtime.taskexecutor.TestGlobalAggregateManager) TestLogger(org.apache.flink.util.TestLogger) TaskEventDispatcher(org.apache.flink.runtime.io.network.TaskEventDispatcher) ResultPartitionConsumableNotifier(org.apache.flink.runtime.io.network.partition.ResultPartitionConsumableNotifier) Matchers.isOneOf(org.hamcrest.Matchers.isOneOf) AbstractInvokable(org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable) KvStateRegistry(org.apache.flink.runtime.query.KvStateRegistry) CheckpointOptions(org.apache.flink.runtime.checkpoint.CheckpointOptions) BroadcastVariableManager(org.apache.flink.runtime.broadcast.BroadcastVariableManager) SerializedValue(org.apache.flink.util.SerializedValue) ResultPartitionDeploymentDescriptor(org.apache.flink.runtime.deployment.ResultPartitionDeploymentDescriptor) Assert.assertFalse(org.junit.Assert.assertFalse) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) AllocationID(org.apache.flink.runtime.clusterframework.types.AllocationID) InputGateDeploymentDescriptor(org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor) Environment(org.apache.flink.runtime.execution.Environment) Mockito.mock(org.mockito.Mockito.mock) OneShotLatch(org.apache.flink.core.testutils.OneShotLatch) MemoryManager(org.apache.flink.runtime.memory.MemoryManager) CheckpointMetaData(org.apache.flink.runtime.checkpoint.CheckpointMetaData) CompletableFuture(java.util.concurrent.CompletableFuture) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) TestingTaskManagerRuntimeInfo(org.apache.flink.runtime.util.TestingTaskManagerRuntimeInfo) TaskMetricGroup(org.apache.flink.runtime.metrics.groups.TaskMetricGroup) TestingClassLoaderLease(org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease) ExternalResourceInfoProvider(org.apache.flink.runtime.externalresource.ExternalResourceInfoProvider) Before(org.junit.Before) NettyShuffleEnvironmentBuilder(org.apache.flink.runtime.io.network.NettyShuffleEnvironmentBuilder) Executor(java.util.concurrent.Executor) Configuration(org.apache.flink.configuration.Configuration) ExecutionState(org.apache.flink.runtime.execution.ExecutionState) FileCache(org.apache.flink.runtime.filecache.FileCache) Test(org.junit.Test) PartitionProducerStateChecker(org.apache.flink.runtime.taskexecutor.PartitionProducerStateChecker) ExecutionAttemptID(org.apache.flink.runtime.executiongraph.ExecutionAttemptID) JobID(org.apache.flink.api.common.JobID) NoOpResultPartitionConsumableNotifier(org.apache.flink.runtime.io.network.partition.NoOpResultPartitionConsumableNotifier) UnregisteredMetricGroups(org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups) TestingUserCodeClassLoader(org.apache.flink.runtime.util.TestingUserCodeClassLoader) TaskInformation(org.apache.flink.runtime.executiongraph.TaskInformation) Collections(java.util.Collections) KvStateRegistry(org.apache.flink.runtime.query.KvStateRegistry) Configuration(org.apache.flink.configuration.Configuration) JobVertexID(org.apache.flink.runtime.jobgraph.JobVertexID) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) KvStateService(org.apache.flink.runtime.taskexecutor.KvStateService) Executor(java.util.concurrent.Executor) TestingTaskManagerRuntimeInfo(org.apache.flink.runtime.util.TestingTaskManagerRuntimeInfo) BroadcastVariableManager(org.apache.flink.runtime.broadcast.BroadcastVariableManager) PartitionProducerStateChecker(org.apache.flink.runtime.taskexecutor.PartitionProducerStateChecker) ResultPartitionConsumableNotifier(org.apache.flink.runtime.io.network.partition.ResultPartitionConsumableNotifier) NoOpResultPartitionConsumableNotifier(org.apache.flink.runtime.io.network.partition.NoOpResultPartitionConsumableNotifier) InputSplitProvider(org.apache.flink.runtime.jobgraph.tasks.InputSplitProvider) JobInformation(org.apache.flink.runtime.executiongraph.JobInformation) TaskInformation(org.apache.flink.runtime.executiongraph.TaskInformation) ExecutionAttemptID(org.apache.flink.runtime.executiongraph.ExecutionAttemptID) IOManager(org.apache.flink.runtime.io.disk.iomanager.IOManager) TaskMetricGroup(org.apache.flink.runtime.metrics.groups.TaskMetricGroup) AllocationID(org.apache.flink.runtime.clusterframework.types.AllocationID) NoOpResultPartitionConsumableNotifier(org.apache.flink.runtime.io.network.partition.NoOpResultPartitionConsumableNotifier) MemoryManager(org.apache.flink.runtime.memory.MemoryManager) TestGlobalAggregateManager(org.apache.flink.runtime.taskexecutor.TestGlobalAggregateManager) FileCache(org.apache.flink.runtime.filecache.FileCache) TestTaskStateManager(org.apache.flink.runtime.state.TestTaskStateManager) TestingClassLoaderLease(org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease) TaskEventDispatcher(org.apache.flink.runtime.io.network.TaskEventDispatcher) JobID(org.apache.flink.api.common.JobID)

Example 2 with TestingClassLoaderLease

use of org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease in project flink by apache.

the class JobMasterServiceLeadershipRunnerTest method testCloseReleasesClassLoaderLease.

@Test
public void testCloseReleasesClassLoaderLease() throws Exception {
    final OneShotLatch closeClassLoaderLeaseLatch = new OneShotLatch();
    final TestingClassLoaderLease classLoaderLease = TestingClassLoaderLease.newBuilder().setCloseRunnable(closeClassLoaderLeaseLatch::trigger).build();
    try (JobManagerRunner jobManagerRunner = newJobMasterServiceLeadershipRunnerBuilder().setClassLoaderLease(classLoaderLease).build()) {
        jobManagerRunner.start();
        jobManagerRunner.close();
        closeClassLoaderLeaseLatch.await();
    }
}
Also used : TestingClassLoaderLease(org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease) OneShotLatch(org.apache.flink.core.testutils.OneShotLatch) Test(org.junit.Test)

Example 3 with TestingClassLoaderLease

use of org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease in project flink by apache.

the class TaskExecutorTest method taskExecutorJobServicesCloseClassLoaderLeaseUponClosing.

@Test
public void taskExecutorJobServicesCloseClassLoaderLeaseUponClosing() throws InterruptedException {
    final OneShotLatch leaseReleaseLatch = new OneShotLatch();
    final OneShotLatch closeHookLatch = new OneShotLatch();
    final TestingClassLoaderLease classLoaderLease = TestingClassLoaderLease.newBuilder().setCloseRunnable(leaseReleaseLatch::trigger).build();
    final TaskExecutor.TaskExecutorJobServices taskExecutorJobServices = TaskExecutor.TaskExecutorJobServices.create(classLoaderLease, closeHookLatch::trigger);
    taskExecutorJobServices.close();
    leaseReleaseLatch.await();
    closeHookLatch.await();
}
Also used : TestingClassLoaderLease(org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease) OneShotLatch(org.apache.flink.core.testutils.OneShotLatch) Test(org.junit.Test)

Aggregations

OneShotLatch (org.apache.flink.core.testutils.OneShotLatch)3 TestingClassLoaderLease (org.apache.flink.runtime.execution.librarycache.TestingClassLoaderLease)3 Test (org.junit.Test)2 Collections (java.util.Collections)1 CompletableFuture (java.util.concurrent.CompletableFuture)1 Executor (java.util.concurrent.Executor)1 Future (java.util.concurrent.Future)1 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)1 JobID (org.apache.flink.api.common.JobID)1 Configuration (org.apache.flink.configuration.Configuration)1 BroadcastVariableManager (org.apache.flink.runtime.broadcast.BroadcastVariableManager)1 CheckpointException (org.apache.flink.runtime.checkpoint.CheckpointException)1 CheckpointMetaData (org.apache.flink.runtime.checkpoint.CheckpointMetaData)1 CheckpointMetricsBuilder (org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder)1 CheckpointOptions (org.apache.flink.runtime.checkpoint.CheckpointOptions)1 AllocationID (org.apache.flink.runtime.clusterframework.types.AllocationID)1 InputGateDeploymentDescriptor (org.apache.flink.runtime.deployment.InputGateDeploymentDescriptor)1 ResultPartitionDeploymentDescriptor (org.apache.flink.runtime.deployment.ResultPartitionDeploymentDescriptor)1 Environment (org.apache.flink.runtime.execution.Environment)1 ExecutionState (org.apache.flink.runtime.execution.ExecutionState)1