Examples with OneShotLatch org.apache.flink.core.testutils.OneShotLatch used on opensource projects

Example 81 with OneShotLatch

use of org.apache.flink.core.testutils.OneShotLatch in project flink by apache.

the class ContinuousFileProcessingMigrationTest method writeReaderSnapshot.

/**
 * Manually run this to write binary snapshot data. Remove @Ignore to run.
 */
@Ignore
@Test
public void writeReaderSnapshot() throws Exception {
    File testFolder = tempFolder.newFolder();
    TimestampedFileInputSplit split1 = new TimestampedFileInputSplit(0, 3, new Path("test/test1"), 0, 100, null);
    TimestampedFileInputSplit split2 = new TimestampedFileInputSplit(10, 2, new Path("test/test2"), 101, 200, null);
    TimestampedFileInputSplit split3 = new TimestampedFileInputSplit(10, 1, new Path("test/test2"), 0, 100, null);
    TimestampedFileInputSplit split4 = new TimestampedFileInputSplit(11, 0, new Path("test/test3"), 0, 100, null);
    // this always blocks to ensure that the reader doesn't to any actual processing so that
    // we keep the state for the four splits
    final OneShotLatch blockingLatch = new OneShotLatch();
    BlockingFileInputFormat format = new BlockingFileInputFormat(blockingLatch, new Path(testFolder.getAbsolutePath()));
    OneInputStreamOperatorTestHarness<TimestampedFileInputSplit, FileInputSplit> testHarness = createHarness(format);
    testHarness.setTimeCharacteristic(TimeCharacteristic.EventTime);
    testHarness.open();
    // create some state in the reader
    testHarness.processElement(new StreamRecord<>(split1));
    testHarness.processElement(new StreamRecord<>(split2));
    testHarness.processElement(new StreamRecord<>(split3));
    testHarness.processElement(new StreamRecord<>(split4));
    // take a snapshot of the operator's state. This will be used
    // to initialize another reader and compare the results of the
    // two operators.
    final OperatorSubtaskState snapshot;
    synchronized (testHarness.getCheckpointLock()) {
        snapshot = testHarness.snapshot(0L, 0L);
    }
    OperatorSnapshotUtil.writeStateHandle(snapshot, "src/test/resources/reader-migration-test-flink" + flinkGenerateSavepointVersion + "-snapshot");
}

Example 82 with OneShotLatch

use of org.apache.flink.core.testutils.OneShotLatch in project flink by apache.

the class JobManagerSharedServicesTest method assertExecutorPoolSize.

private void assertExecutorPoolSize(Executor executor, int expectedPoolSize) throws InterruptedException {
    final CountDownLatch expectedPoolSizeLatch = new CountDownLatch(expectedPoolSize);
    final int expectedPoolSizePlusOne = expectedPoolSize + 1;
    final CountDownLatch expectedPoolSizePlusOneLatch = new CountDownLatch(expectedPoolSizePlusOne);
    final OneShotLatch releaseLatch = new OneShotLatch();
    ThrowingRunnable<Exception> countsDown = () -> {
        expectedPoolSizePlusOneLatch.countDown();
        expectedPoolSizeLatch.countDown();
        // block the runnable to keep the thread occupied
        releaseLatch.await();
    };
    for (int i = 0; i < expectedPoolSizePlusOne; i++) {
        executor.execute(ThrowingRunnable.unchecked(countsDown));
    }
    // the expected pool size latch should complete since we expect to have enough threads
    expectedPoolSizeLatch.await();
    assertEquals(1, expectedPoolSizePlusOneLatch.getCount());
    // unblock the runnables
    releaseLatch.trigger();
}

Example 83 with OneShotLatch

use of org.apache.flink.core.testutils.OneShotLatch in project flink by apache.

the class JobMasterQueryableStateTest method registerSlotsRequiredForJobExecution.

private static void registerSlotsRequiredForJobExecution(JobMasterGateway jobMasterGateway, JobID jobId) throws ExecutionException, InterruptedException {
    final OneShotLatch oneTaskSubmittedLatch = new OneShotLatch();
    final TaskExecutorGateway taskExecutorGateway = new TestingTaskExecutorGatewayBuilder().setSubmitTaskConsumer((taskDeploymentDescriptor, jobMasterId) -> {
        oneTaskSubmittedLatch.trigger();
        return CompletableFuture.completedFuture(Acknowledge.get());
    }).createTestingTaskExecutorGateway();
    JobMasterTestUtils.registerTaskExecutorAndOfferSlots(rpcService, jobMasterGateway, jobId, PARALLELISM, taskExecutorGateway, testingTimeout);
    oneTaskSubmittedLatch.await();
}

Example 84 with OneShotLatch

use of org.apache.flink.core.testutils.OneShotLatch in project flink by apache.

the class JobMasterServiceLeadershipRunnerTest method testCloseReleasesClassLoaderLease.

@Test
public void testCloseReleasesClassLoaderLease() throws Exception {
    final OneShotLatch closeClassLoaderLeaseLatch = new OneShotLatch();
    final TestingClassLoaderLease classLoaderLease = TestingClassLoaderLease.newBuilder().setCloseRunnable(closeClassLoaderLeaseLatch::trigger).build();
    try (JobManagerRunner jobManagerRunner = newJobMasterServiceLeadershipRunnerBuilder().setClassLoaderLease(classLoaderLease).build()) {
        jobManagerRunner.start();
        jobManagerRunner.close();
        closeClassLoaderLeaseLatch.await();
    }
}

Example 85 with OneShotLatch

use of org.apache.flink.core.testutils.OneShotLatch in project flink by apache.

the class BackgroundTaskTest method testRunAfterExecutesBackgroundTaskAfterPreviousHasCompleted.

@Test
public void testRunAfterExecutesBackgroundTaskAfterPreviousHasCompleted() {
    final OneShotLatch blockingLatch = new OneShotLatch();
    final BlockingQueue<Integer> taskCompletions = new ArrayBlockingQueue<>(2);
    final BackgroundTask<Void> backgroundTask = BackgroundTask.initialBackgroundTask(() -> {
        blockingLatch.await();
        taskCompletions.offer(1);
        return null;
    }, TEST_EXECUTOR_RESOURCE.getExecutor()).runAfter(() -> {
        taskCompletions.offer(2);
        return null;
    }, TEST_EXECUTOR_RESOURCE.getExecutor());
    blockingLatch.trigger();
    backgroundTask.getTerminationFuture().join();
    assertThat(taskCompletions, Matchers.contains(1, 2));
}