Search in sources :

Example 1 with HiveQueryNode

use of org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode in project hudi by apache.

the class SimpleWorkflowDagGenerator method build.

@Override
public WorkflowDag build() {
    DagNode root = new InsertNode(DeltaConfig.Config.newBuilder().withNumRecordsToInsert(100).withNumInsertPartitions(1).withNumTimesToRepeat(2).withRecordSize(1000).build());
    DagNode child1 = new InsertNode(DeltaConfig.Config.newBuilder().withNumRecordsToInsert(100).withNumInsertPartitions(1).withNumTimesToRepeat(2).withRecordSize(1000).build());
    root.addChildNode(child1);
    DagNode child1OfChild1 = new UpsertNode(DeltaConfig.Config.newBuilder().withNumRecordsToUpdate(100).withNumUpsertPartitions(2).withNumTimesToRepeat(1).withRecordSize(1000).build());
    // Tests running 2 nodes in parallel
    child1.addChildNode(child1OfChild1);
    List<Pair<String, Integer>> queryAndResult = new ArrayList<>();
    queryAndResult.add(Pair.of("select " + "count(*) from testdb1.table1 group " + "by rider having count(*) < 1", 0));
    DagNode child2OfChild1 = new HiveQueryNode(DeltaConfig.Config.newBuilder().withHiveQueryAndResults(queryAndResult).withHiveLocal(true).build());
    child1.addChildNode(child2OfChild1);
    List<DagNode> rootNodes = new ArrayList<>();
    rootNodes.add(root);
    return new WorkflowDag(rootNodes);
}
Also used : DagNode(org.apache.hudi.integ.testsuite.dag.nodes.DagNode) ArrayList(java.util.ArrayList) HiveQueryNode(org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode) InsertNode(org.apache.hudi.integ.testsuite.dag.nodes.InsertNode) UpsertNode(org.apache.hudi.integ.testsuite.dag.nodes.UpsertNode) Pair(org.apache.hudi.common.util.collection.Pair)

Example 2 with HiveQueryNode

use of org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode in project hudi by apache.

the class HiveSyncDagGenerator method build.

@Override
public WorkflowDag build() {
    DagNode root = new InsertNode(Config.newBuilder().withNumRecordsToInsert(100).withNumInsertPartitions(1).withNumTimesToRepeat(1).withRecordSize(1000).build());
    DagNode child1 = new HiveSyncNode(Config.newBuilder().build());
    root.addChildNode(child1);
    DagNode child2 = new HiveQueryNode(Config.newBuilder().withHiveQueryAndResults(Arrays.asList(Pair.of("select " + "count(*) from testdb1.table1 group " + "by rider having count(*) < 1", 0))).build());
    child1.addChildNode(child2);
    List<DagNode> rootNodes = new ArrayList<>();
    rootNodes.add(root);
    return new WorkflowDag(rootNodes);
}
Also used : DagNode(org.apache.hudi.integ.testsuite.dag.nodes.DagNode) HiveSyncNode(org.apache.hudi.integ.testsuite.dag.nodes.HiveSyncNode) ArrayList(java.util.ArrayList) HiveQueryNode(org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode) InsertNode(org.apache.hudi.integ.testsuite.dag.nodes.InsertNode)

Example 3 with HiveQueryNode

use of org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode in project hudi by apache.

the class HiveSyncDagGeneratorMOR method build.

@Override
public WorkflowDag build() {
    DagNode root = new InsertNode(Config.newBuilder().withNumRecordsToInsert(100).withNumInsertPartitions(1).withNumTimesToRepeat(1).withRecordSize(1000).build());
    DagNode child1 = new HiveSyncNode(Config.newBuilder().withHiveLocal(true).build());
    root.addChildNode(child1);
    DagNode child2 = new HiveQueryNode(Config.newBuilder().withHiveLocal(true).withHiveQueryAndResults(Arrays.asList(Pair.of("select " + "count(*) from testdb1.hive_trips group " + "by rider having count(*) < 1", 0), Pair.of("select " + "count(*) from testdb1.hive_trips ", 100))).withHiveProperties(Arrays.asList("set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat")).build());
    child1.addChildNode(child2);
    List<DagNode> rootNodes = new ArrayList<>();
    rootNodes.add(root);
    return new WorkflowDag(rootNodes);
}
Also used : DagNode(org.apache.hudi.integ.testsuite.dag.nodes.DagNode) HiveSyncNode(org.apache.hudi.integ.testsuite.dag.nodes.HiveSyncNode) ArrayList(java.util.ArrayList) HiveQueryNode(org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode) InsertNode(org.apache.hudi.integ.testsuite.dag.nodes.InsertNode)

Aggregations

ArrayList (java.util.ArrayList)3 DagNode (org.apache.hudi.integ.testsuite.dag.nodes.DagNode)3 HiveQueryNode (org.apache.hudi.integ.testsuite.dag.nodes.HiveQueryNode)3 InsertNode (org.apache.hudi.integ.testsuite.dag.nodes.InsertNode)3 HiveSyncNode (org.apache.hudi.integ.testsuite.dag.nodes.HiveSyncNode)2 Pair (org.apache.hudi.common.util.collection.Pair)1 UpsertNode (org.apache.hudi.integ.testsuite.dag.nodes.UpsertNode)1