Examples with BlockManager org.apache.spark.storage.BlockManager used on opensource projects

Example 1 with BlockManager

use of org.apache.spark.storage.BlockManager in project beam by apache.

the class GlobalWatermarkHolder method clear.

@VisibleForTesting
public static synchronized void clear() {
    sourceTimes.clear();
    lastWatermarkedBatchTime = 0;
    writeLocalWatermarkCopy(null);
    final SparkEnv sparkEnv = SparkEnv.get();
    if (sparkEnv != null) {
        final BlockManager blockManager = sparkEnv.blockManager();
        blockManager.removeBlock(WATERMARKS_BLOCK_ID, true);
    }
}

Example 2 with BlockManager

use of org.apache.spark.storage.BlockManager in project beam by apache.

the class GlobalWatermarkHolder method advance.

/**
 * Advances the watermarks to the next-in-line watermarks. SparkWatermarks are monotonically
 * increasing.
 */
private static void advance(final String batchId) {
    synchronized (GlobalWatermarkHolder.class) {
        final BlockManager blockManager = SparkEnv.get().blockManager();
        final Map<Integer, SparkWatermarks> newWatermarks = computeNewWatermarks(blockManager);
        if (!newWatermarks.isEmpty()) {
            writeRemoteWatermarkBlock(newWatermarks, blockManager);
            writeLocalWatermarkCopy(newWatermarks);
        } else {
            LOG.info("No new watermarks could be computed upon completion of batch: {}", batchId);
        }
    }
}