Search in sources :

Example 1 with CompactorRequestTypeInfo

use of org.apache.flink.connector.file.sink.compactor.operator.CompactorRequestTypeInfo in project flink by apache.

the class FileSink method addPreCommitTopology.

@Override
public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
    FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
    if (strategy == null) {
        // not enabled, handlers will be added to process the remaining states of the compact
        // coordinator and the compactor operators.
        SingleOutputStreamOperator<Either<CommittableMessage<FileSinkCommittable>, CompactorRequest>> coordinatorOp = committableStream.forward().transform("CompactorCoordinator", new EitherTypeInfo<>(committableStream.getType(), new CompactorRequestTypeInfo(bucketsBuilder::getCommittableSerializer)), new CompactCoordinatorStateHandlerFactory(bucketsBuilder::getCommittableSerializer)).setParallelism(committableStream.getParallelism()).uid("FileSinkCompactorCoordinator");
        return coordinatorOp.forward().transform("CompactorOperator", committableStream.getType(), new CompactorOperatorStateHandlerFactory(bucketsBuilder::getCommittableSerializer, bucketsBuilder::createBucketWriter)).setParallelism(committableStream.getParallelism()).uid("FileSinkCompactorOperator");
    }
    // explicitly rebalance here is required, or the partitioner will be forward, which is in
    // fact the partitioner from the writers to the committers
    SingleOutputStreamOperator<CompactorRequest> coordinatorOp = committableStream.rebalance().transform("CompactorCoordinator", new CompactorRequestTypeInfo(bucketsBuilder::getCommittableSerializer), new CompactCoordinatorFactory(strategy, bucketsBuilder::getCommittableSerializer)).setParallelism(1).uid("FileSinkCompactorCoordinator");
    // parallelism of the compactors is not configurable at present, since it must be identical
    // to that of the committers, or the committable summary and the committables may be
    // distributed to different committers, which will cause a failure
    TypeInformation<CommittableMessage<FileSinkCommittable>> committableType = committableStream.getType();
    return coordinatorOp.transform("CompactorOperator", committableType, new CompactorOperatorFactory(strategy, bucketsBuilder.getFileCompactor(), bucketsBuilder::getCommittableSerializer, bucketsBuilder::createBucketWriter)).setParallelism(committableStream.getParallelism()).uid("FileSinkCompactorOperator");
}
Also used : CommittableMessage(org.apache.flink.streaming.api.connector.sink2.CommittableMessage) CompactorOperatorFactory(org.apache.flink.connector.file.sink.compactor.operator.CompactorOperatorFactory) CompactCoordinatorFactory(org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinatorFactory) FileCompactStrategy(org.apache.flink.connector.file.sink.compactor.FileCompactStrategy) CompactorOperatorStateHandlerFactory(org.apache.flink.connector.file.sink.compactor.operator.CompactorOperatorStateHandlerFactory) CompactorRequestTypeInfo(org.apache.flink.connector.file.sink.compactor.operator.CompactorRequestTypeInfo) CompactCoordinatorStateHandlerFactory(org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinatorStateHandlerFactory) Either(org.apache.flink.types.Either) CompactorRequest(org.apache.flink.connector.file.sink.compactor.operator.CompactorRequest)

Aggregations

FileCompactStrategy (org.apache.flink.connector.file.sink.compactor.FileCompactStrategy)1 CompactCoordinatorFactory (org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinatorFactory)1 CompactCoordinatorStateHandlerFactory (org.apache.flink.connector.file.sink.compactor.operator.CompactCoordinatorStateHandlerFactory)1 CompactorOperatorFactory (org.apache.flink.connector.file.sink.compactor.operator.CompactorOperatorFactory)1 CompactorOperatorStateHandlerFactory (org.apache.flink.connector.file.sink.compactor.operator.CompactorOperatorStateHandlerFactory)1 CompactorRequest (org.apache.flink.connector.file.sink.compactor.operator.CompactorRequest)1 CompactorRequestTypeInfo (org.apache.flink.connector.file.sink.compactor.operator.CompactorRequestTypeInfo)1 CommittableMessage (org.apache.flink.streaming.api.connector.sink2.CommittableMessage)1 Either (org.apache.flink.types.Either)1