Examples with UnsafeInmemoryMergeHolder org.apache.carbondata.processing.loading.sort.unsafe.holder.UnsafeInmemoryMergeHolder used on opensource projects

Example 1 with UnsafeInmemoryMergeHolder

use of org.apache.carbondata.processing.loading.sort.unsafe.holder.UnsafeInmemoryMergeHolder in project carbondata by apache.

the class UnsafeInMemoryIntermediateDataMerger method getSortedRecordFromMemory.

/**
 * This method will be used to get the sorted record from file
 *
 * @return sorted record sorted record
 * @throws CarbonSortKeyAndGroupByException
 */
private UnsafeCarbonRowForMerge getSortedRecordFromMemory() throws CarbonSortKeyAndGroupByException {
    UnsafeCarbonRowForMerge row = null;
    // poll the top object from heap
    // heap maintains binary tree which is based on heap condition that will
    // be based on comparator we are passing the heap
    // when will call poll it will always delete root of the tree and then
    // it does trickel down operation complexity is log(n)
    UnsafeInmemoryMergeHolder poll = this.recordHolderHeap.poll();
    // get the row from chunk
    row = poll.getRow();
    // check if there no entry present
    if (!poll.hasNext()) {
        // change the file counter
        --this.holderCounter;
        // reaturn row
        return row;
    }
    // read new row
    poll.readRow();
    // add to heap
    this.recordHolderHeap.add(poll);
    // return row
    return row;
}

Example 2 with UnsafeInmemoryMergeHolder

use of org.apache.carbondata.processing.loading.sort.unsafe.holder.UnsafeInmemoryMergeHolder in project carbondata by apache.

the class UnsafeInMemoryIntermediateDataMerger method startSorting.

/**
 * Below method will be used to start storing process This method will get
 * all the temp files present in sort temp folder then it will create the
 * record holder heap and then it will read first record from each file and
 * initialize the heap
 *
 * @throws CarbonSortKeyAndGroupByException
 */
private void startSorting() throws CarbonSortKeyAndGroupByException {
    LOGGER.info("Number of row pages in intermediate merger: " + this.holderCounter);
    // create record holder heap
    createRecordHolderQueue(unsafeCarbonRowPages);
    // iterate over file list and create chunk holder and add to heap
    LOGGER.info("Started adding first record from row page");
    UnsafeInmemoryMergeHolder unsafePageHolder = null;
    byte index = 0;
    for (UnsafeCarbonRowPage unsafeCarbonRowPage : unsafeCarbonRowPages) {
        // create chunk holder
        unsafePageHolder = new UnsafeInmemoryMergeHolder(unsafeCarbonRowPage, index++);
        // initialize
        unsafePageHolder.readRow();
        // add to heap
        this.recordHolderHeap.add(unsafePageHolder);
    }
    LOGGER.info("Heap Size" + this.recordHolderHeap.size());
}