Examples with SelectionVector4 org.apache.drill.exec.record.selection.SelectionVector4 used on opensource projects

Example 16 with SelectionVector4

use of org.apache.drill.exec.record.selection.SelectionVector4 in project drill by apache.

the class MSortTemplate method setup.

@Override
public void setup(final FragmentContext context, final BufferAllocator allocator, final SelectionVector4 vector4, final VectorContainer hyperBatch) throws SchemaChangeException {
    // we pass in the local hyperBatch since that is where we'll be reading data.
    Preconditions.checkNotNull(vector4);
    this.vector4 = vector4.createNewWrapperCurrent();
    this.context = context;
    vector4.clear();
    doSetup(context, hyperBatch, null);
    runStarts.add(0);
    int batch = 0;
    final int totalCount = this.vector4.getTotalCount();
    for (int i = 0; i < totalCount; i++) {
        final int newBatch = this.vector4.get(i) >>> 16;
        if (newBatch == batch) {
            continue;
        } else if (newBatch == batch + 1) {
            runStarts.add(i);
            batch = newBatch;
        } else {
            throw new UnsupportedOperationException(String.format("Missing batch. batch: %d newBatch: %d", batch, newBatch));
        }
    }
    final DrillBuf drillBuf = allocator.buffer(4 * totalCount);
    try {
        desiredRecordBatchCount = context.getConfig().getInt(ExecConstants.EXTERNAL_SORT_MSORT_MAX_BATCHSIZE);
    } catch (ConfigException.Missing e) {
        // value not found, use default value instead
        desiredRecordBatchCount = Character.MAX_VALUE;
    }
    aux = new SelectionVector4(drillBuf, totalCount, desiredRecordBatchCount);
}

Example 17 with SelectionVector4

use of org.apache.drill.exec.record.selection.SelectionVector4 in project drill by apache.

the class MSortTemplate method sort.

/**
   * Sort (really, merge) a set of pre-sorted runs to produce a combined
   * result set. Merging is done in the selection vector, record data does
   * not move.
   * <p>
   * Runs are merge pairwise in multiple passes, providing performance
   * of O(n * m * log(n)), where n = number of runs, m = number of records
   * per run.
   */
@Override
public void sort(final VectorContainer container) {
    while (runStarts.size() > 1) {
        final int totalCount = this.vector4.getTotalCount();
        // check if we're cancelled/failed recently
        if (!context.shouldContinue()) {
            return;
        }
        int outIndex = 0;
        final Queue<Integer> newRunStarts = Queues.newLinkedBlockingQueue();
        newRunStarts.add(outIndex);
        final int size = runStarts.size();
        for (int i = 0; i < size / 2; i++) {
            final int left = runStarts.poll();
            final int right = runStarts.poll();
            Integer end = runStarts.peek();
            if (end == null) {
                end = totalCount;
            }
            outIndex = merge(left, right, end, outIndex);
            if (outIndex < vector4.getTotalCount()) {
                newRunStarts.add(outIndex);
            }
        }
        if (outIndex < totalCount) {
            copyRun(outIndex, totalCount);
        }
        @SuppressWarnings("resource") final SelectionVector4 tmp = aux.createNewWrapperCurrent(desiredRecordBatchCount);
        aux.clear();
        aux = vector4.createNewWrapperCurrent(desiredRecordBatchCount);
        vector4.clear();
        vector4 = tmp.createNewWrapperCurrent(desiredRecordBatchCount);
        tmp.clear();
        runStarts = newRunStarts;
    }
    aux.clear();
}

Example 18 with SelectionVector4

use of org.apache.drill.exec.record.selection.SelectionVector4 in project drill by apache.

the class PriorityQueueCopierTemplate method setup.

@Override
public void setup(FragmentContext context, BufferAllocator allocator, VectorAccessible hyperBatch, List<BatchGroup> batchGroups, VectorAccessible outgoing) throws SchemaChangeException {
    this.hyperBatch = hyperBatch;
    this.batchGroups = batchGroups;
    this.outgoing = outgoing;
    this.size = batchGroups.size();
    @SuppressWarnings("resource") final DrillBuf drillBuf = allocator.buffer(4 * size);
    vector4 = new SelectionVector4(drillBuf, size, Character.MAX_VALUE);
    doSetup(context, hyperBatch, outgoing);
    queueSize = 0;
    for (int i = 0; i < size; i++) {
        int index = batchGroups.get(i).getNextIndex();
        vector4.set(i, i, index);
        if (index > -1) {
            siftUp();
            queueSize++;
        }
    }
}

Example 19 with SelectionVector4

use of org.apache.drill.exec.record.selection.SelectionVector4 in project drill by apache.

the class PriorityQueueCopierTemplate method setup.

@Override
public void setup(FragmentContext context, BufferAllocator allocator, VectorAccessible hyperBatch, List<BatchGroup> batchGroups, VectorAccessible outgoing) throws SchemaChangeException {
    this.hyperBatch = hyperBatch;
    this.batchGroups = batchGroups;
    this.outgoing = outgoing;
    this.size = batchGroups.size();
    final DrillBuf drillBuf = allocator.buffer(4 * size);
    vector4 = new SelectionVector4(drillBuf, size, Character.MAX_VALUE);
    doSetup(context, hyperBatch, outgoing);
    queueSize = 0;
    for (int i = 0; i < size; i++) {
        vector4.set(i, i, batchGroups.get(i).getNextIndex());
        siftUp();
        queueSize++;
    }
}