Search in sources :

Example 1 with RangePartitionerImpl

use of org.apache.carbondata.processing.loading.partition.impl.RangePartitionerImpl in project carbondata by apache.

the class DataConverterProcessorStepImpl method initializeSortColumnRangesPartitioner.

/**
 * initialize partitioner for sort column ranges
 */
private void initializeSortColumnRangesPartitioner() {
    // convert user specified sort-column ranges
    SortColumnRangeInfo sortColumnRangeInfo = configuration.getSortColumnRangeInfo();
    int rangeValueCnt = sortColumnRangeInfo.getUserSpecifiedRanges().length;
    CarbonRow[] convertedSortColumnRanges = new CarbonRow[rangeValueCnt];
    for (int i = 0; i < rangeValueCnt; i++) {
        Object[] fakeOriginRow = new Object[configuration.getDataFields().length];
        String[] oneBound = StringUtils.splitPreserveAllTokens(sortColumnRangeInfo.getUserSpecifiedRanges()[i], sortColumnRangeInfo.getSeparator(), -1);
        // set the corresponding sort column
        int j = 0;
        for (int colIdx : sortColumnRangeInfo.getSortColumnIndex()) {
            fakeOriginRow[colIdx] = oneBound[j++];
        }
        CarbonRow fakeCarbonRow = new CarbonRow(fakeOriginRow);
        convertFakeRow(fakeCarbonRow, sortColumnRangeInfo);
        convertedSortColumnRanges[i] = fakeCarbonRow;
    }
    // sort the range bounds (sort in carbon is a little different from what we think)
    Arrays.sort(convertedSortColumnRanges, new RawRowComparator(sortColumnRangeInfo.getSortColumnIndex(), sortColumnRangeInfo.getIsSortColumnNoDict()));
    // range partitioner to dispatch rows by sort columns
    this.partitioner = new RangePartitionerImpl(convertedSortColumnRanges, new RawRowComparator(sortColumnRangeInfo.getSortColumnIndex(), sortColumnRangeInfo.getIsSortColumnNoDict()));
}
Also used : RawRowComparator(org.apache.carbondata.processing.loading.partition.impl.RawRowComparator) CarbonRow(org.apache.carbondata.core.datastore.row.CarbonRow) SortColumnRangeInfo(org.apache.carbondata.core.metadata.schema.SortColumnRangeInfo) RangePartitionerImpl(org.apache.carbondata.processing.loading.partition.impl.RangePartitionerImpl)

Aggregations

CarbonRow (org.apache.carbondata.core.datastore.row.CarbonRow)1 SortColumnRangeInfo (org.apache.carbondata.core.metadata.schema.SortColumnRangeInfo)1 RangePartitionerImpl (org.apache.carbondata.processing.loading.partition.impl.RangePartitionerImpl)1 RawRowComparator (org.apache.carbondata.processing.loading.partition.impl.RawRowComparator)1