Search in sources :

Example 1 with SimpleVersionedSerializerTypeSerializerProxy

use of org.apache.flink.core.io.SimpleVersionedSerializerTypeSerializerProxy in project flink by apache.

the class CompactCoordinatorTest method testStateHandler.

@Test
public void testStateHandler() throws Exception {
    FileCompactStrategy strategy = Builder.newBuilder().setSizeThreshold(10).build();
    CompactCoordinator coordinator = new CompactCoordinator(strategy, getTestCommittableSerializer());
    // with . prefix
    FileSinkCommittable committable0 = committable("0", ".0", 5);
    FileSinkCommittable committable1 = committable("0", ".1", 6);
    // without . prefix
    FileSinkCommittable committable2 = committable("0", "2", 6);
    OperatorSubtaskState state;
    try (OneInputStreamOperatorTestHarness<CommittableMessage<FileSinkCommittable>, CompactorRequest> harness = new OneInputStreamOperatorTestHarness<>(coordinator)) {
        harness.setup();
        harness.open();
        harness.processElement(message(committable0));
        Assert.assertEquals(0, harness.extractOutputValues().size());
        harness.prepareSnapshotPreBarrier(1);
        state = harness.snapshot(1, 1);
    }
    CompactCoordinatorStateHandler handler = new CompactCoordinatorStateHandler(getTestCommittableSerializer());
    try (OneInputStreamOperatorTestHarness<CommittableMessage<FileSinkCommittable>, Either<CommittableMessage<FileSinkCommittable>, CompactorRequest>> harness = new OneInputStreamOperatorTestHarness<>(handler)) {
        harness.setup(new EitherSerializer<>(new SimpleVersionedSerializerTypeSerializerProxy<>(() -> new CommittableMessageSerializer<>(getTestCommittableSerializer())), new SimpleVersionedSerializerTypeSerializerProxy<>(() -> new CompactorRequestSerializer(getTestCommittableSerializer()))));
        harness.initializeState(state);
        harness.open();
        Assert.assertEquals(1, harness.extractOutputValues().size());
        harness.processElement(message(committable1));
        harness.processElement(message(committable2));
        List<Either<CommittableMessage<FileSinkCommittable>, CompactorRequest>> results = harness.extractOutputValues();
        Assert.assertEquals(3, results.size());
        // restored request
        Assert.assertTrue(results.get(0).isRight());
        assertToCompact(results.get(0).right(), committable0);
        // committable with . prefix should also be passed through
        Assert.assertTrue(results.get(1).isLeft() && results.get(1).left() instanceof CommittableWithLineage);
        Assert.assertEquals(((CommittableWithLineage<FileSinkCommittable>) results.get(1).left()).getCommittable(), committable1);
        // committable without . prefix should be passed through normally
        Assert.assertTrue(results.get(2).isLeft() && results.get(2).left() instanceof CommittableWithLineage);
        Assert.assertEquals(((CommittableWithLineage<FileSinkCommittable>) results.get(2).left()).getCommittable(), committable2);
    }
}
Also used : CommittableMessage(org.apache.flink.streaming.api.connector.sink2.CommittableMessage) CommittableWithLineage(org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage) CompactCoordinatorStateHandler(org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinatorStateHandler) OneInputStreamOperatorTestHarness(org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness) CompactCoordinator(org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinator) OperatorSubtaskState(org.apache.flink.runtime.checkpoint.OperatorSubtaskState) SimpleVersionedSerializerTypeSerializerProxy(org.apache.flink.core.io.SimpleVersionedSerializerTypeSerializerProxy) FileSinkCommittable(org.apache.flink.connector.file.sink.FileSinkCommittable) Either(org.apache.flink.types.Either) CompactorRequest(org.apache.flink.connector.file.sink.compactor.operator.CompactorRequest) CompactorRequestSerializer(org.apache.flink.connector.file.sink.compactor.operator.CompactorRequestSerializer) Test(org.junit.Test)

Aggregations

FileSinkCommittable (org.apache.flink.connector.file.sink.FileSinkCommittable)1 CompactCoordinator (org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinator)1 CompactCoordinatorStateHandler (org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinatorStateHandler)1 CompactorRequest (org.apache.flink.connector.file.sink.compactor.operator.CompactorRequest)1 CompactorRequestSerializer (org.apache.flink.connector.file.sink.compactor.operator.CompactorRequestSerializer)1 SimpleVersionedSerializerTypeSerializerProxy (org.apache.flink.core.io.SimpleVersionedSerializerTypeSerializerProxy)1 OperatorSubtaskState (org.apache.flink.runtime.checkpoint.OperatorSubtaskState)1 CommittableMessage (org.apache.flink.streaming.api.connector.sink2.CommittableMessage)1 CommittableWithLineage (org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage)1 OneInputStreamOperatorTestHarness (org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness)1 Either (org.apache.flink.types.Either)1 Test (org.junit.Test)1