Search in sources :

Example 1 with RealtimeSink

use of co.cask.cdap.etl.api.realtime.RealtimeSink in project cdap by caskdata.

the class ETLWorker method destroy.

@Override
public void destroy() {
    Destroyables.destroyQuietly(source);
    Destroyables.destroyQuietly(transformExecutor);
    for (RealtimeSink sink : sinks.values()) {
        Destroyables.destroyQuietly(sink);
    }
}
Also used : RealtimeSink(co.cask.cdap.etl.api.realtime.RealtimeSink)

Example 2 with RealtimeSink

use of co.cask.cdap.etl.api.realtime.RealtimeSink in project cdap by caskdata.

the class ETLWorker method initializeSinks.

@SuppressWarnings("unchecked")
private void initializeSinks(WorkerContext context, Map<String, TransformDetail> transformationMap, PipelinePhase pipeline) throws Exception {
    Set<StageInfo> sinkInfos = pipeline.getStagesOfType(RealtimeSink.PLUGIN_TYPE);
    sinks = new HashMap<>(sinkInfos.size());
    for (StageInfo sinkInfo : sinkInfos) {
        String sinkName = sinkInfo.getName();
        RealtimeSink sink = context.newPluginInstance(sinkName);
        sink = new LoggedRealtimeSink(sinkName, sink);
        WorkerRealtimeContext sinkContext = new WorkerRealtimeContext(context, metrics, new TxLookupProvider(context), sinkInfo);
        LOG.debug("Sink Class : {}", sink.getClass().getName());
        sink.initialize(sinkContext);
        sink = new TrackedRealtimeSink(sink, new DefaultStageMetrics(metrics, sinkName));
        Transformation identityTransformation = new Transformation() {

            @Override
            public void transform(Object input, Emitter emitter) throws Exception {
                emitter.emit(input);
            }
        };
        // we use identity transformation to simplify executing transformation in pipeline (similar to ETLMapreduce),
        // since we want to emit metrics during write to sink and not during this transformation, we use NoOpMetrics.
        TrackedTransform trackedTransform = new TrackedTransform(identityTransformation, new DefaultStageMetrics(metrics, sinkName), TrackedTransform.RECORDS_IN, null, context.getDataTracer(sinkName));
        transformationMap.put(sinkInfo.getName(), new TransformDetail(trackedTransform, new HashSet<String>()));
        sinks.put(sinkInfo.getName(), sink);
    }
}
Also used : TrackedTransform(co.cask.cdap.etl.common.TrackedTransform) Transformation(co.cask.cdap.etl.api.Transformation) Emitter(co.cask.cdap.etl.api.Emitter) DefaultEmitter(co.cask.cdap.etl.common.DefaultEmitter) TrackedEmitter(co.cask.cdap.etl.common.TrackedEmitter) StageInfo(co.cask.cdap.etl.planner.StageInfo) TxLookupProvider(co.cask.cdap.etl.common.TxLookupProvider) TransformDetail(co.cask.cdap.etl.common.TransformDetail) RealtimeSink(co.cask.cdap.etl.api.realtime.RealtimeSink) DefaultStageMetrics(co.cask.cdap.etl.common.DefaultStageMetrics) HashSet(java.util.HashSet)

Aggregations

RealtimeSink (co.cask.cdap.etl.api.realtime.RealtimeSink)2 Emitter (co.cask.cdap.etl.api.Emitter)1 Transformation (co.cask.cdap.etl.api.Transformation)1 DefaultEmitter (co.cask.cdap.etl.common.DefaultEmitter)1 DefaultStageMetrics (co.cask.cdap.etl.common.DefaultStageMetrics)1 TrackedEmitter (co.cask.cdap.etl.common.TrackedEmitter)1 TrackedTransform (co.cask.cdap.etl.common.TrackedTransform)1 TransformDetail (co.cask.cdap.etl.common.TransformDetail)1 TxLookupProvider (co.cask.cdap.etl.common.TxLookupProvider)1 StageInfo (co.cask.cdap.etl.planner.StageInfo)1 HashSet (java.util.HashSet)1