Search in sources :

Example 1 with Index

use of org.apache.carbondata.hadoop.internal.index.Index in project carbondata by apache.

the class IndexedSegment method getSplits.

@Override
public List<InputSplit> getSplits(JobContext job, FilterResolverIntf filterResolver) throws IOException {
    // do as following
    // 1. create the index or get from cache by the filter name in the configuration
    // 2. filter by index to get the filtered block
    // 3. create input split from filtered block
    List<InputSplit> output = new LinkedList<>();
    Index index = loader.load(this);
    List<Block> blocks = index.filter(job, filterResolver);
    for (Block block : blocks) {
        output.add(makeInputSplit(block));
    }
    return output;
}
Also used : Block(org.apache.carbondata.hadoop.internal.index.Block) Index(org.apache.carbondata.hadoop.internal.index.Index) InputSplit(org.apache.hadoop.mapreduce.InputSplit) LinkedList(java.util.LinkedList)

Aggregations

LinkedList (java.util.LinkedList)1 Block (org.apache.carbondata.hadoop.internal.index.Block)1 Index (org.apache.carbondata.hadoop.internal.index.Index)1 InputSplit (org.apache.hadoop.mapreduce.InputSplit)1