Search in sources :

Example 1 with WriterContext

use of org.apache.hudi.integ.testsuite.dag.WriterContext in project hudi by apache.

the class HoodieTestSuiteJob method runTestSuite.

public void runTestSuite() {
    try {
        WorkflowDag workflowDag = createWorkflowDag();
        log.info("Workflow Dag => " + DagUtils.convertDagToYaml(workflowDag));
        long startTime = System.currentTimeMillis();
        WriterContext writerContext = new WriterContext(jsc, props, cfg, keyGenerator, sparkSession);
        writerContext.initContext(jsc);
        startOtherServicesIfNeeded(writerContext);
        if (this.cfg.saferSchemaEvolution) {
            // rollback most recent upsert/insert, by default.
            int numRollbacks = 2;
            // if root is RollbackNode, get num_rollbacks
            List<DagNode> root = workflowDag.getNodeList();
            if (!root.isEmpty() && root.get(0) instanceof RollbackNode) {
                numRollbacks = root.get(0).getConfig().getNumRollbacks();
            }
            int version = getSchemaVersionFromCommit(numRollbacks - 1);
            SaferSchemaDagScheduler dagScheduler = new SaferSchemaDagScheduler(workflowDag, writerContext, jsc, version);
            dagScheduler.schedule();
        } else {
            DagScheduler dagScheduler = new DagScheduler(workflowDag, writerContext, jsc);
            dagScheduler.schedule();
        }
        log.info("Finished scheduling all tasks, Time taken {}", System.currentTimeMillis() - startTime);
    } catch (Exception e) {
        log.error("Failed to run Test Suite ", e);
        throw new HoodieException("Failed to run Test Suite ", e);
    } finally {
        stopQuietly();
    }
}
Also used : WriterContext(org.apache.hudi.integ.testsuite.dag.WriterContext) DagNode(org.apache.hudi.integ.testsuite.dag.nodes.DagNode) SaferSchemaDagScheduler(org.apache.hudi.integ.testsuite.dag.scheduler.SaferSchemaDagScheduler) DagScheduler(org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler) RollbackNode(org.apache.hudi.integ.testsuite.dag.nodes.RollbackNode) SaferSchemaDagScheduler(org.apache.hudi.integ.testsuite.dag.scheduler.SaferSchemaDagScheduler) WorkflowDag(org.apache.hudi.integ.testsuite.dag.WorkflowDag) HoodieException(org.apache.hudi.exception.HoodieException) HoodieException(org.apache.hudi.exception.HoodieException) IOException(java.io.IOException)

Aggregations

IOException (java.io.IOException)1 HoodieException (org.apache.hudi.exception.HoodieException)1 WorkflowDag (org.apache.hudi.integ.testsuite.dag.WorkflowDag)1 WriterContext (org.apache.hudi.integ.testsuite.dag.WriterContext)1 DagNode (org.apache.hudi.integ.testsuite.dag.nodes.DagNode)1 RollbackNode (org.apache.hudi.integ.testsuite.dag.nodes.RollbackNode)1 DagScheduler (org.apache.hudi.integ.testsuite.dag.scheduler.DagScheduler)1 SaferSchemaDagScheduler (org.apache.hudi.integ.testsuite.dag.scheduler.SaferSchemaDagScheduler)1