Search in sources :

Example 1 with MaterializableIterable

use of org.apache.crunch.materialize.MaterializableIterable in project crunch by cloudera.

the class MRPipeline method run.

@Override
public PipelineResult run() {
    MSCRPlanner planner = new MSCRPlanner(this, outputTargets);
    PipelineResult res = null;
    try {
        res = planner.plan(jarClass, conf).execute();
    } catch (IOException e) {
        LOG.error(e);
        return PipelineResult.EMPTY;
    }
    for (PCollectionImpl<?> c : outputTargets.keySet()) {
        if (outputTargetsToMaterialize.containsKey(c)) {
            MaterializableIterable iter = outputTargetsToMaterialize.get(c);
            iter.materialize();
            c.materializeAt(iter.getSourceTarget());
            outputTargetsToMaterialize.remove(c);
        } else {
            boolean materialized = false;
            for (Target t : outputTargets.get(c)) {
                if (!materialized && t instanceof Source) {
                    c.materializeAt((SourceTarget) t);
                    materialized = true;
                }
            }
        }
    }
    outputTargets.clear();
    return res;
}
Also used : SourceTarget(org.apache.crunch.SourceTarget) Target(org.apache.crunch.Target) ReadableSourceTarget(org.apache.crunch.io.ReadableSourceTarget) MSCRPlanner(org.apache.crunch.impl.mr.plan.MSCRPlanner) PipelineResult(org.apache.crunch.PipelineResult) IOException(java.io.IOException) Source(org.apache.crunch.Source) TableSource(org.apache.crunch.TableSource) MaterializableIterable(org.apache.crunch.materialize.MaterializableIterable)

Aggregations

IOException (java.io.IOException)1 PipelineResult (org.apache.crunch.PipelineResult)1 Source (org.apache.crunch.Source)1 SourceTarget (org.apache.crunch.SourceTarget)1 TableSource (org.apache.crunch.TableSource)1 Target (org.apache.crunch.Target)1 MSCRPlanner (org.apache.crunch.impl.mr.plan.MSCRPlanner)1 ReadableSourceTarget (org.apache.crunch.io.ReadableSourceTarget)1 MaterializableIterable (org.apache.crunch.materialize.MaterializableIterable)1