Search in sources :

Example 1 with CollectSinkOperator

use of org.apache.flink.streaming.api.operators.collect.CollectSinkOperator in project flink by apache.

the class DataStream method executeAndCollectWithClient.

ClientAndIterator<T> executeAndCollectWithClient(String jobExecutionName) throws Exception {
    TypeSerializer<T> serializer = getType().createSerializer(getExecutionEnvironment().getConfig());
    String accumulatorName = "dataStreamCollect_" + UUID.randomUUID().toString();
    StreamExecutionEnvironment env = getExecutionEnvironment();
    CollectSinkOperatorFactory<T> factory = new CollectSinkOperatorFactory<>(serializer, accumulatorName);
    CollectSinkOperator<T> operator = (CollectSinkOperator<T>) factory.getOperator();
    CollectResultIterator<T> iterator = new CollectResultIterator<>(operator.getOperatorIdFuture(), serializer, accumulatorName, env.getCheckpointConfig());
    CollectStreamSink<T> sink = new CollectStreamSink<>(this, factory);
    sink.name("Data stream collect sink");
    env.addOperator(sink.getTransformation());
    final JobClient jobClient = env.executeAsync(jobExecutionName);
    iterator.setJobClient(jobClient);
    return new ClientAndIterator<>(jobClient, iterator);
}
Also used : ClientAndIterator(org.apache.flink.streaming.api.operators.collect.ClientAndIterator) CollectResultIterator(org.apache.flink.streaming.api.operators.collect.CollectResultIterator) CollectStreamSink(org.apache.flink.streaming.api.operators.collect.CollectStreamSink) JobClient(org.apache.flink.core.execution.JobClient) CollectSinkOperator(org.apache.flink.streaming.api.operators.collect.CollectSinkOperator) CollectSinkOperatorFactory(org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)

Example 2 with CollectSinkOperator

use of org.apache.flink.streaming.api.operators.collect.CollectSinkOperator in project flink by apache.

the class CollectDynamicSink method getSinkRuntimeProvider.

@Override
public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
    return (DataStreamSinkProvider) (providerContext, inputStream) -> {
        final CheckpointConfig checkpointConfig = inputStream.getExecutionEnvironment().getCheckpointConfig();
        final ExecutionConfig config = inputStream.getExecutionConfig();
        final TypeSerializer<RowData> externalSerializer = InternalTypeInfo.<RowData>of(consumedDataType.getLogicalType()).createSerializer(config);
        final String accumulatorName = tableIdentifier.getObjectName();
        final CollectSinkOperatorFactory<RowData> factory = new CollectSinkOperatorFactory<>(externalSerializer, accumulatorName, maxBatchSize, socketTimeout);
        final CollectSinkOperator<RowData> operator = (CollectSinkOperator<RowData>) factory.getOperator();
        iterator = new CollectResultIterator<>(operator.getOperatorIdFuture(), externalSerializer, accumulatorName, checkpointConfig);
        converter = context.createDataStructureConverter(consumedDataType);
        converter.open(RuntimeConverter.Context.create(classLoader));
        final CollectStreamSink<RowData> sink = new CollectStreamSink<>(inputStream, factory);
        providerContext.generateUid(COLLECT_TRANSFORMATION).ifPresent(sink::uid);
        return sink.name("Collect table sink");
    };
}
Also used : CheckpointConfig(org.apache.flink.streaming.api.environment.CheckpointConfig) DataStreamSinkProvider(org.apache.flink.table.connector.sink.DataStreamSinkProvider) ExecutionConfig(org.apache.flink.api.common.ExecutionConfig) CollectStreamSink(org.apache.flink.streaming.api.operators.collect.CollectStreamSink) RowData(org.apache.flink.table.data.RowData) CollectSinkOperator(org.apache.flink.streaming.api.operators.collect.CollectSinkOperator) CollectSinkOperatorFactory(org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory)

Aggregations

CollectSinkOperator (org.apache.flink.streaming.api.operators.collect.CollectSinkOperator)2 CollectSinkOperatorFactory (org.apache.flink.streaming.api.operators.collect.CollectSinkOperatorFactory)2 CollectStreamSink (org.apache.flink.streaming.api.operators.collect.CollectStreamSink)2 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)1 JobClient (org.apache.flink.core.execution.JobClient)1 CheckpointConfig (org.apache.flink.streaming.api.environment.CheckpointConfig)1 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)1 ClientAndIterator (org.apache.flink.streaming.api.operators.collect.ClientAndIterator)1 CollectResultIterator (org.apache.flink.streaming.api.operators.collect.CollectResultIterator)1 DataStreamSinkProvider (org.apache.flink.table.connector.sink.DataStreamSinkProvider)1 RowData (org.apache.flink.table.data.RowData)1