Examples with TransExecutionConfiguration org.pentaho.di.trans.TransExecutionConfiguration used on opensource projects

Example 11 with TransExecutionConfiguration

use of org.pentaho.di.trans.TransExecutionConfiguration in project pentaho-kettle by pentaho.

the class MasterSlaveIT method runParallelFileReadOnSlavesWithPartitioning2.

/**
 * This test reads a CSV file in parallel on all 3 slaves, each with 4 partitions.<br>
 * This is a variation on the test right above, with 2 steps in sequence in clustering & partitioning.<br>
 * It then passes the data over to a dummy step on the slaves.<br>
 */
public void runParallelFileReadOnSlavesWithPartitioning2() throws Exception {
    TransMeta transMeta = loadTransMetaReplaceSlavesInCluster(clusterGenerator, "test/org/pentaho/di/cluster/test-parallel-file-read-on-slaves-with-partitioning2.ktr");
    TransExecutionConfiguration config = createClusteredTransExecutionConfiguration();
    TransSplitter transSplitter = Trans.executeClustered(transMeta, config);
    LogChannel logChannel = createLogChannel("cluster unit test <runParallelFileReadOnSlavesWithPartitioning2>");
    long nrErrors = Trans.monitorClusteredTransformation(logChannel, transSplitter, null, 1);
    assertEquals(0L, nrErrors);
    String result = loadFileContent(transMeta, "${java.io.tmpdir}/test-parallel-file-read-on-slaves-with-partitioning2.txt");
    assertEqualsIgnoreWhitespacesAndCase("100", result);
}

Example 12 with TransExecutionConfiguration

use of org.pentaho.di.trans.TransExecutionConfiguration in project pentaho-kettle by pentaho.

the class MasterSlaveIT method runParallelFileReadOnSlavesWithPartitioning.

/**
 * This test reads a CSV file in parallel on all 3 slaves, each with 4 partitions.<br>
 * It then passes the data over to a dummy step on the slaves.<br>
 */
public void runParallelFileReadOnSlavesWithPartitioning() throws Exception {
    TransMeta transMeta = loadTransMetaReplaceSlavesInCluster(clusterGenerator, "test/org/pentaho/di/cluster/test-parallel-file-read-on-slaves-with-partitioning.ktr");
    TransExecutionConfiguration config = createClusteredTransExecutionConfiguration();
    TransSplitter transSplitter = Trans.executeClustered(transMeta, config);
    LogChannel logChannel = createLogChannel("cluster unit test <runParallelFileReadOnSlavesWithPartitioning>");
    long nrErrors = Trans.monitorClusteredTransformation(logChannel, transSplitter, null, 1);
    assertEquals(0L, nrErrors);
    String result = loadFileContent(transMeta, "${java.io.tmpdir}/test-parallel-file-read-on-slaves-with-partitioning.txt");
    assertEqualsIgnoreWhitespacesAndCase("100", result);
}

Example 13 with TransExecutionConfiguration

use of org.pentaho.di.trans.TransExecutionConfiguration in project pentaho-kettle by pentaho.

the class MasterSlaveIT method runOneStepClustered.

/**
 * This test generates rows on the master, generates random values clustered and brings them back the master.<br>
 * See also: PDI-6324 : Generate Rows to a clustered step ceases to work
 */
public void runOneStepClustered() throws Exception {
    TransMeta transMeta = loadTransMetaReplaceSlavesInCluster(clusterGenerator, "test/org/pentaho/di/cluster/one-step-clustered.ktr");
    TransExecutionConfiguration config = createClusteredTransExecutionConfiguration();
    TransSplitter transSplitter = Trans.executeClustered(transMeta, config);
    LogChannel logChannel = createLogChannel("cluster unit test <runOneStepClustered>");
    long nrErrors = Trans.monitorClusteredTransformation(logChannel, transSplitter, null, 1);
    assertEquals(0L, nrErrors);
    String result = loadFileContent(transMeta, "${java.io.tmpdir}/one-step-clustered.txt");
    assertEqualsIgnoreWhitespacesAndCase("10000", result);
}

Example 14 with TransExecutionConfiguration

use of org.pentaho.di.trans.TransExecutionConfiguration in project pentaho-kettle by pentaho.

the class MasterSlaveIT method runSubtransformationClustered.

/**
 * This test check passing rows to sub-transformation executed on cluster
 * See PDI-10704 for details
 * @throws Exception
 */
public void runSubtransformationClustered() throws Exception {
    TransMeta transMeta = loadTransMetaReplaceSlavesInCluster(clusterGenerator, "test/org/pentaho/di/cluster/test-subtrans-clustered.ktr");
    TransExecutionConfiguration config = createClusteredTransExecutionConfiguration();
    Result prevResult = new Result();
    prevResult.setRows(getSampleRows());
    config.setPreviousResult(prevResult);
    TransSplitter transSplitter = Trans.executeClustered(transMeta, config);
    LogChannel logChannel = createLogChannel("cluster unit test <runSubtransformationClustered>");
    long nrErrors = Trans.monitorClusteredTransformation(logChannel, transSplitter, null, 1);
    assertEquals(0L, nrErrors);
    String result = loadFileContent(transMeta, "${java.io.tmpdir}/test-subtrans-clustered.txt");
    assertEqualsIgnoreWhitespacesAndCase("10", result);
}

Example 15 with TransExecutionConfiguration

use of org.pentaho.di.trans.TransExecutionConfiguration in project pentaho-kettle by pentaho.

the class MasterSlaveIT method runMultipleCopiesOnMultipleSlaves.

/**
 * This test reads a CSV file and sends the data to 3 copies on 3 slave servers.<br>
 */
public void runMultipleCopiesOnMultipleSlaves() throws Exception {
    TransMeta transMeta = loadTransMetaReplaceSlavesInCluster(clusterGenerator, "test/org/pentaho/di/cluster/test-multiple-copies-on-multiple-slaves.ktr");
    TransExecutionConfiguration config = createClusteredTransExecutionConfiguration();
    TransSplitter transSplitter = Trans.executeClustered(transMeta, config);
    LogChannel logChannel = createLogChannel("cluster unit test <testMultipleCopiesOnMultipleSlaves>");
    long nrErrors = Trans.monitorClusteredTransformation(logChannel, transSplitter, null, 1);
    assertEquals(0L, nrErrors);
    String result = loadFileContent(transMeta, "${java.io.tmpdir}/test-multiple-copies-on-multiple-slaves.txt");
    assertEqualsIgnoreWhitespacesAndCase("100", result);
}