Examples with SpilledRun org.apache.drill.exec.physical.impl.xsort.managed.BatchGroup.SpilledRun used on opensource projects

Example 1 with SpilledRun

use of org.apache.drill.exec.physical.impl.xsort.managed.BatchGroup.SpilledRun in project drill by apache.

the class ExternalSortBatch method mergeRuns.

private void mergeRuns(int targetCount) {
    // Determine the number of runs to merge. The count should be the
    // target count. However, to prevent possible memory overrun, we
    // double-check with actual spill batch size and only spill as much
    // as fits in the merge memory pool.
    int mergeCount = 0;
    long mergeSize = 0;
    for (SpilledRun run : spilledRuns) {
        long batchSize = run.getBatchSize();
        if (mergeSize + batchSize > mergeMemoryPool) {
            break;
        }
        mergeSize += batchSize;
        mergeCount++;
        if (mergeCount == targetCount) {
            break;
        }
    }
    // Must always spill at least 2, even if this creates an over-size
    // spill file. But, if this is a final consolidation, we may have only
    // a single batch.
    mergeCount = Math.max(mergeCount, 2);
    mergeCount = Math.min(mergeCount, spilledRuns.size());
    // Do the actual spill.
    mergeAndSpill(spilledRuns, mergeCount);
}