Examples with ManagedStateSpillableStateStore org.apache.apex.malhar.lib.state.spillable.managed.ManagedStateSpillableStateStore used on opensource projects

Example 1 with ManagedStateSpillableStateStore

use of org.apache.apex.malhar.lib.state.spillable.managed.ManagedStateSpillableStateStore in project apex-malhar by apache.

the class SpillableDSBenchmarkTest method testSpillableMap.

@Test
public void testSpillableMap() {
    byte[] ID1 = new byte[] { (byte) 1 };
    ManagedStateSpillableStateStore store = new ManagedStateSpillableStateStore();
    ((TFileImpl.DTFileImpl) store.getFileAccess()).setBasePath("target/temp");
    StringSerde keySerde = createKeySerde();
    Serde<String> valueSerde = createValueSerde();
    SpillableMapImpl<String, String> map = new SpillableMapImpl<String, String>(store, ID1, 0L, keySerde, valueSerde);
    store.setup(testMeta.operatorContext);
    map.setup(testMeta.operatorContext);
    final long startTime = System.currentTimeMillis();
    long windowId = 0;
    store.beginWindow(++windowId);
    map.beginWindow(windowId);
    int outputTimes = 0;
    for (int i = 0; i < loopCount; ++i) {
        putEntry(map);
        if (i % tuplesPerWindow == 0) {
            map.endWindow();
            store.endWindow();
            if (i % (tuplesPerWindow * checkPointWindows) == 0) {
                store.beforeCheckpoint(windowId);
                if (windowId > commitDelays) {
                    store.committed(windowId - commitDelays);
                }
            }
            // next window
            store.beginWindow(++windowId);
            map.beginWindow(windowId);
        }
        long spentTime = System.currentTimeMillis() - startTime;
        if (spentTime > outputTimes * 5000) {
            ++outputTimes;
            logger.info("Total Statistics: Spent {} mills for {} operation. average/second: {}", spentTime, i, i * 1000 / spentTime);
            checkEnvironment();
        }
    }
    long spentTime = System.currentTimeMillis() - startTime;
    logger.info("Spent {} mills for {} operation. average: {}", spentTime, loopCount, loopCount / spentTime);
}

Example 2 with ManagedStateSpillableStateStore

use of org.apache.apex.malhar.lib.state.spillable.managed.ManagedStateSpillableStateStore in project apex-malhar by apache.

the class KeyedWindowedOperatorBenchmarkApp method createUpdatedDataStorage.

protected SpillableSetMultimapImpl<Window, String> createUpdatedDataStorage(Configuration conf, SpillableComplexComponentImpl sccImpl) {
    String basePath = getStoreBasePath(conf);
    ManagedStateSpillableStateStore store = new ManagedStateSpillableStateStore();
    ((TFileImpl.DTFileImpl) store.getFileAccess()).setBasePath(basePath);
    SpillableSetMultimapImpl<Window, String> dataStorage = new SpillableSetMultimapImpl<Window, String>(store, new byte[] { (byte) 1 }, 0, new GenericSerde<Window>(), new GenericSerde<String>());
    return dataStorage;
}

Example 3 with ManagedStateSpillableStateStore

use of org.apache.apex.malhar.lib.state.spillable.managed.ManagedStateSpillableStateStore in project apex-malhar by apache.

the class SpillableBenchmarkApp method createStore.

public ManagedStateSpillableStateStore createStore(Configuration conf) {
    String basePath = getStoreBasePath(conf);
    ManagedStateSpillableStateStore store = new ManagedStateSpillableStateStore();
    ((TFileImpl.DTFileImpl) store.getFileAccess()).setBasePath(basePath);
    return store;
}