Examples with InputBatch org.apache.drill.exec.physical.impl.xsort.managed.BatchGroup.InputBatch used on opensource projects

Example 1 with InputBatch

use of org.apache.drill.exec.physical.impl.xsort.managed.BatchGroup.InputBatch in project drill by apache.

the class ExternalSortBatch method spillFromMemory.

/**
   * This operator has accumulated a set of sorted incoming record batches.
   * We wish to spill some of them to disk. To do this, a "copier"
   * merges the target batches to produce a stream of new (merged) batches
   * which are then written to disk.
   * <p>
   * This method spills only half the accumulated batches
   * minimizing unnecessary disk writes. The exact count must lie between
   * the minimum and maximum spill counts.
   */
private void spillFromMemory() {
    // Determine the number of batches to spill to create a spill file
    // of the desired size. The actual file size might be a bit larger
    // or smaller than the target, which is expected.
    int spillCount = 0;
    long spillSize = 0;
    for (InputBatch batch : bufferedBatches) {
        long batchSize = batch.getDataSize();
        spillSize += batchSize;
        spillCount++;
        if (spillSize + batchSize / 2 > spillFileSize) {
            break;
        }
    }
    // Must always spill at least 2, even if this creates an over-size
    // spill file. But, if this is a final consolidation, we may have only
    // a single batch.
    spillCount = Math.max(spillCount, 2);
    spillCount = Math.min(spillCount, bufferedBatches.size());
    // Do the actual spill.
    mergeAndSpill(bufferedBatches, spillCount);
}

Example 2 with InputBatch

use of org.apache.drill.exec.physical.impl.xsort.managed.BatchGroup.InputBatch in project drill by axbaretto.

the class BufferedBatches method prepareSpill.

public List<BatchGroup> prepareSpill(long targetSpillSize) {
    // Determine the number of batches to spill to create a spill file
    // of the desired size. The actual file size might be a bit larger
    // or smaller than the target, which is expected.
    int spillCount = 0;
    long spillSize = 0;
    for (InputBatch batch : bufferedBatches) {
        long batchSize = batch.getDataSize();
        spillSize += batchSize;
        spillCount++;
        if (spillSize + batchSize / 2 > targetSpillSize) {
            break;
        }
    }
    // Must always spill at least 2, even if this creates an over-size
    // spill file. But, if this is a final consolidation, we may have only
    // a single batch.
    spillCount = Math.max(spillCount, 2);
    spillCount = Math.min(spillCount, bufferedBatches.size());
    return SpilledRuns.prepareSpillBatches(bufferedBatches, spillCount);
}