Search in sources :

Example 1 with Dependency

use of org.apache.spark.Dependency in project hive by apache.

the class SparkUtilities method rddToString.

private static void rddToString(RDD rdd, StringBuilder sb, String offset) {
    sb.append(offset).append(rdd.getClass().getCanonicalName()).append("[").append(rdd.hashCode()).append("]");
    if (rdd.getStorageLevel().useMemory()) {
        sb.append("(cached)");
    }
    sb.append("\n");
    Collection<Dependency> dependencies = JavaConversions.asJavaCollection(rdd.dependencies());
    if (dependencies != null) {
        offset += "\t";
        for (Dependency dependency : dependencies) {
            RDD parentRdd = dependency.rdd();
            rddToString(parentRdd, sb, offset);
        }
    } else if (rdd instanceof UnionRDD) {
        UnionRDD unionRDD = (UnionRDD) rdd;
        offset += "\t";
        Collection<RDD> parentRdds = JavaConversions.asJavaCollection(unionRDD.rdds());
        for (RDD parentRdd : parentRdds) {
            rddToString(parentRdd, sb, offset);
        }
    }
}
Also used : UnionRDD(org.apache.spark.rdd.UnionRDD) JavaPairRDD(org.apache.spark.api.java.JavaPairRDD) RDD(org.apache.spark.rdd.RDD) Collection(java.util.Collection) Dependency(org.apache.spark.Dependency) UnionRDD(org.apache.spark.rdd.UnionRDD)

Aggregations

Collection (java.util.Collection)1 Dependency (org.apache.spark.Dependency)1 JavaPairRDD (org.apache.spark.api.java.JavaPairRDD)1 RDD (org.apache.spark.rdd.RDD)1 UnionRDD (org.apache.spark.rdd.UnionRDD)1