Search in sources :

Example 1 with StringDistributionMerger

use of org.apache.druid.indexing.common.task.batch.parallel.distribution.StringDistributionMerger in project druid by druid-io.

the class ParallelIndexSupervisorTask method determineRangePartition.

private PartitionBoundaries determineRangePartition(Collection<StringDistribution> distributions) {
    StringDistributionMerger distributionMerger = new StringSketchMerger();
    distributions.forEach(distributionMerger::merge);
    StringDistribution mergedDistribution = distributionMerger.getResult();
    DimensionRangePartitionsSpec partitionsSpec = (DimensionRangePartitionsSpec) ingestionSchema.getTuningConfig().getGivenOrDefaultPartitionsSpec();
    final PartitionBoundaries partitions;
    Integer targetRowsPerSegment = partitionsSpec.getTargetRowsPerSegment();
    if (targetRowsPerSegment == null) {
        partitions = mergedDistribution.getEvenPartitionsByMaxSize(partitionsSpec.getMaxRowsPerSegment());
    } else {
        partitions = mergedDistribution.getEvenPartitionsByTargetSize(targetRowsPerSegment);
    }
    return partitions;
}
Also used : AtomicInteger(java.util.concurrent.atomic.AtomicInteger) StringDistribution(org.apache.druid.indexing.common.task.batch.parallel.distribution.StringDistribution) StringSketchMerger(org.apache.druid.indexing.common.task.batch.parallel.distribution.StringSketchMerger) StringDistributionMerger(org.apache.druid.indexing.common.task.batch.parallel.distribution.StringDistributionMerger) DimensionRangePartitionsSpec(org.apache.druid.indexer.partitions.DimensionRangePartitionsSpec) PartitionBoundaries(org.apache.druid.timeline.partition.PartitionBoundaries)

Aggregations

AtomicInteger (java.util.concurrent.atomic.AtomicInteger)1 DimensionRangePartitionsSpec (org.apache.druid.indexer.partitions.DimensionRangePartitionsSpec)1 StringDistribution (org.apache.druid.indexing.common.task.batch.parallel.distribution.StringDistribution)1 StringDistributionMerger (org.apache.druid.indexing.common.task.batch.parallel.distribution.StringDistributionMerger)1 StringSketchMerger (org.apache.druid.indexing.common.task.batch.parallel.distribution.StringSketchMerger)1 PartitionBoundaries (org.apache.druid.timeline.partition.PartitionBoundaries)1