Search in sources :

Example 1 with NumberedOverwritePartialShardSpec

use of org.apache.druid.timeline.partition.NumberedOverwritePartialShardSpec in project druid by druid-io.

the class IndexerSQLMetadataStorageCoordinatorTest method testAllocatePendingSegmentsWithOvershadowingSegments.

@Test
public void testAllocatePendingSegmentsWithOvershadowingSegments() throws IOException {
    final String dataSource = "ds";
    final Interval interval = Intervals.of("2017-01-01/2017-02-01");
    String prevSegmentId = null;
    for (int i = 0; i < 10; i++) {
        final SegmentIdWithShardSpec identifier = coordinator.allocatePendingSegment(dataSource, "seq", prevSegmentId, interval, new NumberedOverwritePartialShardSpec(0, 1, (short) (i + 1)), "version", false);
        Assert.assertEquals(StringUtils.format("ds_2017-01-01T00:00:00.000Z_2017-02-01T00:00:00.000Z_version%s", "_" + (i + PartitionIds.NON_ROOT_GEN_START_PARTITION_ID)), identifier.toString());
        prevSegmentId = identifier.toString();
        final Set<DataSegment> toBeAnnounced = Collections.singleton(new DataSegment(identifier.getDataSource(), identifier.getInterval(), identifier.getVersion(), null, Collections.emptyList(), Collections.emptyList(), ((NumberedOverwriteShardSpec) identifier.getShardSpec()).withAtomicUpdateGroupSize(1), 0, 10L));
        final Set<DataSegment> announced = coordinator.announceHistoricalSegments(toBeAnnounced);
        Assert.assertEquals(toBeAnnounced, announced);
    }
    final Collection<DataSegment> visibleSegments = coordinator.retrieveUsedSegmentsForInterval(dataSource, interval, Segments.ONLY_VISIBLE);
    Assert.assertEquals(1, visibleSegments.size());
    Assert.assertEquals(new DataSegment(dataSource, interval, "version", null, Collections.emptyList(), Collections.emptyList(), new NumberedOverwriteShardSpec(9 + PartitionIds.NON_ROOT_GEN_START_PARTITION_ID, 0, 1, (short) 9, (short) 1), 0, 10L), Iterables.getOnlyElement(visibleSegments));
}
Also used : NumberedOverwritePartialShardSpec(org.apache.druid.timeline.partition.NumberedOverwritePartialShardSpec) NumberedOverwriteShardSpec(org.apache.druid.timeline.partition.NumberedOverwriteShardSpec) SegmentIdWithShardSpec(org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec) DataSegment(org.apache.druid.timeline.DataSegment) Interval(org.joda.time.Interval) Test(org.junit.Test)

Example 2 with NumberedOverwritePartialShardSpec

use of org.apache.druid.timeline.partition.NumberedOverwritePartialShardSpec in project druid by druid-io.

the class TaskLockboxTest method testRequestForNewSegmentWithSegmentLock.

@Test
public void testRequestForNewSegmentWithSegmentLock() {
    final Task task = NoopTask.create();
    lockbox.add(task);
    allocateSegmentsAndAssert(task, "seq", 3, NumberedPartialShardSpec.instance());
    allocateSegmentsAndAssert(task, "seq2", 2, new NumberedOverwritePartialShardSpec(0, 3, (short) 1));
    final List<TaskLock> locks = lockbox.findLocksForTask(task);
    Assert.assertEquals(5, locks.size());
    int expectedPartitionId = 0;
    for (TaskLock lock : locks) {
        Assert.assertTrue(lock instanceof SegmentLock);
        final SegmentLock segmentLock = (SegmentLock) lock;
        Assert.assertEquals(expectedPartitionId++, segmentLock.getPartitionId());
        if (expectedPartitionId == 3) {
            expectedPartitionId = PartitionIds.NON_ROOT_GEN_START_PARTITION_ID;
        }
    }
}
Also used : AbstractTask(org.apache.druid.indexing.common.task.AbstractTask) Task(org.apache.druid.indexing.common.task.Task) NoopTask(org.apache.druid.indexing.common.task.NoopTask) NumberedOverwritePartialShardSpec(org.apache.druid.timeline.partition.NumberedOverwritePartialShardSpec) TaskLock(org.apache.druid.indexing.common.TaskLock) SegmentLock(org.apache.druid.indexing.common.SegmentLock) Test(org.junit.Test)

Aggregations

NumberedOverwritePartialShardSpec (org.apache.druid.timeline.partition.NumberedOverwritePartialShardSpec)2 Test (org.junit.Test)2 SegmentLock (org.apache.druid.indexing.common.SegmentLock)1 TaskLock (org.apache.druid.indexing.common.TaskLock)1 AbstractTask (org.apache.druid.indexing.common.task.AbstractTask)1 NoopTask (org.apache.druid.indexing.common.task.NoopTask)1 Task (org.apache.druid.indexing.common.task.Task)1 SegmentIdWithShardSpec (org.apache.druid.segment.realtime.appenderator.SegmentIdWithShardSpec)1 DataSegment (org.apache.druid.timeline.DataSegment)1 NumberedOverwriteShardSpec (org.apache.druid.timeline.partition.NumberedOverwriteShardSpec)1 Interval (org.joda.time.Interval)1