Search in sources :

Example 36 with Connection

use of co.cask.cdap.etl.proto.Connection in project cdap by caskdata.

the class Dag method subsetAround.

/**
 * Return a subset of this dag starting from the specified stage, without going past any node in the
 * child stop nodes and parent stop nodes. If the parent or child stop nodes contain the starting stage, it
 * will be ignored.
 * This is equivalent to taking the nodes from {@link #accessibleFrom(Set, Set)}, {@link #parentsOf(String, Set)},
 * and building a dag from them.
 *
 * @param stage the stage to start at
 * @param childStopNodes set of nodes to stop traversing forwards on
 * @param parentStopNodes set of nodes to stop traversing backwards on
 * @return a dag created from the stages given and child nodes of those stages and parent nodes of those stages.
 */
public Dag subsetAround(String stage, Set<String> childStopNodes, Set<String> parentStopNodes) {
    Set<String> nodes = Sets.union(accessibleFrom(stage, childStopNodes), parentsOf(stage, parentStopNodes));
    Set<Connection> connections = new HashSet<>();
    for (String node : nodes) {
        for (String outputNode : outgoingConnections.get(node)) {
            if (nodes.contains(outputNode)) {
                connections.add(new Connection(node, outputNode));
            }
        }
    }
    return new Dag(connections);
}
Also used : Connection(co.cask.cdap.etl.proto.Connection) HashSet(java.util.HashSet)

Aggregations

Connection (co.cask.cdap.etl.proto.Connection)36 Test (org.junit.Test)26 HashSet (java.util.HashSet)23 HashMap (java.util.HashMap)10 StageSpec (co.cask.cdap.etl.spec.StageSpec)8 PipelinePhase (co.cask.cdap.etl.common.PipelinePhase)6 PipelineSpec (co.cask.cdap.etl.spec.PipelineSpec)5 Resources (co.cask.cdap.api.Resources)4 UpgradeContext (co.cask.cdap.etl.proto.UpgradeContext)4 ArrayList (java.util.ArrayList)4 ArtifactSelectorConfig (co.cask.cdap.etl.proto.ArtifactSelectorConfig)3 Dag (co.cask.cdap.etl.planner.Dag)2 Plugin (co.cask.cdap.etl.proto.v1.Plugin)2 ETLPlugin (co.cask.cdap.etl.proto.v2.ETLPlugin)2 ImmutableSet (com.google.common.collect.ImmutableSet)2 Map (java.util.Map)2 Set (java.util.Set)2 Schema (co.cask.cdap.api.data.schema.Schema)1 PartitionedFileSet (co.cask.cdap.api.dataset.lib.PartitionedFileSet)1 ConditionBranches (co.cask.cdap.etl.planner.ConditionBranches)1