Search in sources :

Example 1 with CarbonSortBatch

use of org.apache.carbondata.processing.loading.row.CarbonSortBatch in project carbondata by apache.

the class UnsafeBatchParallelReadMergeSorterImpl method sort.

@Override
public Iterator<CarbonRowBatch>[] sort(Iterator<CarbonRowBatch>[] iterators) throws CarbonDataLoadingException {
    this.executorService = Executors.newFixedThreadPool(iterators.length);
    this.threadStatusObserver = new ThreadStatusObserver(this.executorService);
    int batchSize = CarbonProperties.getInstance().getBatchSize();
    final SortBatchHolder sortBatchHolder = new SortBatchHolder(sortParameters, iterators.length, this.threadStatusObserver);
    try {
        for (int i = 0; i < iterators.length; i++) {
            executorService.execute(new SortIteratorThread(iterators[i], sortBatchHolder, batchSize, rowCounter, this.threadStatusObserver));
        }
    } catch (Exception e) {
        checkError();
        throw new CarbonDataLoadingException("Problem while shutdown the server ", e);
    }
    checkError();
    // Creates the iterator to read from merge sorter.
    Iterator<CarbonSortBatch> batchIterator = new CarbonIterator<CarbonSortBatch>() {

        @Override
        public boolean hasNext() {
            return sortBatchHolder.hasNext();
        }

        @Override
        public CarbonSortBatch next() {
            return new CarbonSortBatch(sortBatchHolder.next());
        }
    };
    return new Iterator[] { batchIterator };
}
Also used : CarbonDataLoadingException(org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException) CarbonIterator(org.apache.carbondata.common.CarbonIterator) CarbonIterator(org.apache.carbondata.common.CarbonIterator) Iterator(java.util.Iterator) CarbonSortBatch(org.apache.carbondata.processing.loading.row.CarbonSortBatch) CarbonSortKeyAndGroupByException(org.apache.carbondata.processing.sort.exception.CarbonSortKeyAndGroupByException) MemoryException(org.apache.carbondata.core.memory.MemoryException) CarbonDataWriterException(org.apache.carbondata.core.datastore.exception.CarbonDataWriterException) CarbonDataLoadingException(org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException)

Aggregations

Iterator (java.util.Iterator)1 CarbonIterator (org.apache.carbondata.common.CarbonIterator)1 CarbonDataWriterException (org.apache.carbondata.core.datastore.exception.CarbonDataWriterException)1 MemoryException (org.apache.carbondata.core.memory.MemoryException)1 CarbonDataLoadingException (org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException)1 CarbonSortBatch (org.apache.carbondata.processing.loading.row.CarbonSortBatch)1 CarbonSortKeyAndGroupByException (org.apache.carbondata.processing.sort.exception.CarbonSortKeyAndGroupByException)1