Examples with FSWindowDataManager org.apache.apex.malhar.lib.wal.FSWindowDataManager used on opensource projects

Example 26 with FSWindowDataManager

use of org.apache.apex.malhar.lib.wal.FSWindowDataManager in project apex-malhar by apache.

the class FileSplitterTest method testIdempotencyWithBlocksThreshold.

@Test
public void testIdempotencyWithBlocksThreshold() throws InterruptedException {
    FSWindowDataManager fsWindowDataManager = new FSWindowDataManager();
    testMeta.fileSplitter.setWindowDataManager(fsWindowDataManager);
    testMeta.fileSplitter.setBlocksThreshold(10);
    testMeta.fileSplitter.scanner.setScanIntervalMillis(500);
    testMeta.fileSplitter.setup(testMeta.context);
    testBlocksThreshold();
    testMeta.fileMetadataSink.clear();
    testMeta.blockMetadataSink.clear();
    testMeta.fileSplitter.setup(testMeta.context);
    for (int i = 1; i < 8; i++) {
        testMeta.fileSplitter.beginWindow(i);
    }
    Assert.assertEquals("Files", 12, testMeta.fileMetadataSink.collectedTuples.size());
    Assert.assertEquals("Blocks", 62, testMeta.blockMetadataSink.collectedTuples.size());
}

Example 27 with FSWindowDataManager

use of org.apache.apex.malhar.lib.wal.FSWindowDataManager in project apex-malhar by apache.

the class ExactlyOnceJdbcOutputApp method populateDAG.

@Override
public void populateDAG(DAG dag, Configuration conf) {
    KafkaSinglePortStringInputOperator kafkaInput = dag.addOperator("kafkaInput", new KafkaSinglePortStringInputOperator());
    kafkaInput.setWindowDataManager(new FSWindowDataManager());
    UniqueCounterFlat count = dag.addOperator("count", new UniqueCounterFlat());
    CountStoreOperator store = dag.addOperator("store", new CountStoreOperator());
    store.setStore(new JdbcTransactionalStore());
    ConsoleOutputOperator cons = dag.addOperator("console", new ConsoleOutputOperator());
    dag.addStream("words", kafkaInput.outputPort, count.data);
    dag.addStream("counts", count.counts, store.input, cons.input);
}

Example 28 with FSWindowDataManager

use of org.apache.apex.malhar.lib.wal.FSWindowDataManager in project apex-malhar by apache.

the class AbstractUpsertOutputOperator method setup.

// end of input port implementation
@Override
public void setup(Context.OperatorContext context) {
    super.setup(context);
    windowDataManager = getWindowDataManager();
    if (windowDataManager == null) {
        windowDataManager = new FSWindowDataManager();
    }
    windowDataManager.setup(context);
}