Search in sources :

Example 6 with MaxSizeSplitHintSpec

use of org.apache.druid.data.input.MaxSizeSplitHintSpec in project druid by druid-io.

the class LocalInputSourceTest method testCreateSplitsRespectingSplitHintSpec.

@Test
public void testCreateSplitsRespectingSplitHintSpec() {
    final long fileSize = 15;
    final HumanReadableBytes maxSplitSize = new HumanReadableBytes(50L);
    final List<File> files = mockFiles(10, fileSize);
    final LocalInputSource inputSource = new LocalInputSource(null, null, files);
    final List<InputSplit<List<File>>> splits = inputSource.createSplits(new NoopInputFormat(), new MaxSizeSplitHintSpec(maxSplitSize, null)).collect(Collectors.toList());
    Assert.assertEquals(4, splits.size());
    Assert.assertEquals(3, splits.get(0).get().size());
    Assert.assertEquals(3, splits.get(1).get().size());
    Assert.assertEquals(3, splits.get(2).get().size());
    Assert.assertEquals(1, splits.get(3).get().size());
}
Also used : HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) File(java.io.File) InputSplit(org.apache.druid.data.input.InputSplit) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) Test(org.junit.Test)

Example 7 with MaxSizeSplitHintSpec

use of org.apache.druid.data.input.MaxSizeSplitHintSpec in project druid by druid-io.

the class CombiningInputSourceTest method testEstimateNumSplits.

@Test
public void testEstimateNumSplits() {
    final File file = EasyMock.niceMock(File.class);
    EasyMock.expect(file.length()).andReturn(5L).anyTimes();
    EasyMock.replay(file);
    final TestFileInputSource fileSource = new TestFileInputSource(generateFiles(3));
    final TestUriInputSource uriInputSource = new TestUriInputSource(ImmutableList.of(URI.create("http://test.com/http-test1"), URI.create("http://test.com/http-test2"), URI.create("http://test.com/http-test3")));
    final CombiningInputSource combiningInputSource = new CombiningInputSource(ImmutableList.of(fileSource, uriInputSource));
    Assert.assertEquals(combiningInputSource.estimateNumSplits(new NoopInputFormat(), new MaxSizeSplitHintSpec(new HumanReadableBytes(5L), null)), 6);
}
Also used : HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) File(java.io.File) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) Test(org.junit.Test)

Example 8 with MaxSizeSplitHintSpec

use of org.apache.druid.data.input.MaxSizeSplitHintSpec in project druid by druid-io.

the class OssInputSourceTest method testCreateSplitsWithSplitHintSpecRespectingHint.

@Test
public void testCreateSplitsWithSplitHintSpecRespectingHint() {
    EasyMock.reset(OSSCLIENT);
    expectListObjects(PREFIXES.get(0), ImmutableList.of(EXPECTED_URIS.get(0)), CONTENT);
    expectListObjects(PREFIXES.get(1), ImmutableList.of(EXPECTED_URIS.get(1)), CONTENT);
    EasyMock.replay(OSSCLIENT);
    OssInputSource inputSource = new OssInputSource(OSSCLIENT, INPUT_DATA_CONFIG, null, PREFIXES, null, null);
    Stream<InputSplit<List<CloudObjectLocation>>> splits = inputSource.createSplits(new JsonInputFormat(JSONPathSpec.DEFAULT, null, null), new MaxSizeSplitHintSpec(new HumanReadableBytes(CONTENT.length * 3L), null));
    Assert.assertEquals(ImmutableList.of(EXPECTED_URIS.stream().map(CloudObjectLocation::new).collect(Collectors.toList())), splits.map(InputSplit::get).collect(Collectors.toList()));
    EasyMock.verify(OSSCLIENT);
}
Also used : JsonInputFormat(org.apache.druid.data.input.impl.JsonInputFormat) CloudObjectLocation(org.apache.druid.data.input.impl.CloudObjectLocation) HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) InputSplit(org.apache.druid.data.input.InputSplit) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Example 9 with MaxSizeSplitHintSpec

use of org.apache.druid.data.input.MaxSizeSplitHintSpec in project druid by druid-io.

the class OssInputSourceTest method testWithPrefixesSplit.

@Test
public void testWithPrefixesSplit() {
    EasyMock.reset(OSSCLIENT);
    expectListObjects(PREFIXES.get(0), ImmutableList.of(EXPECTED_URIS.get(0)), CONTENT);
    expectListObjects(PREFIXES.get(1), ImmutableList.of(EXPECTED_URIS.get(1)), CONTENT);
    EasyMock.replay(OSSCLIENT);
    OssInputSource inputSource = new OssInputSource(OSSCLIENT, INPUT_DATA_CONFIG, null, PREFIXES, null, null);
    Stream<InputSplit<List<CloudObjectLocation>>> splits = inputSource.createSplits(new JsonInputFormat(JSONPathSpec.DEFAULT, null, null), new MaxSizeSplitHintSpec(null, 1));
    Assert.assertEquals(EXPECTED_COORDS, splits.map(InputSplit::get).collect(Collectors.toList()));
    EasyMock.verify(OSSCLIENT);
}
Also used : JsonInputFormat(org.apache.druid.data.input.impl.JsonInputFormat) CloudObjectLocation(org.apache.druid.data.input.impl.CloudObjectLocation) InputSplit(org.apache.druid.data.input.InputSplit) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Example 10 with MaxSizeSplitHintSpec

use of org.apache.druid.data.input.MaxSizeSplitHintSpec in project druid by druid-io.

the class GoogleCloudStorageInputSourceTest method testWithPrefixesSplit.

@Test
public void testWithPrefixesSplit() throws IOException {
    EasyMock.reset(STORAGE);
    EasyMock.reset(INPUT_DATA_CONFIG);
    addExpectedPrefixObjects(PREFIXES.get(0), ImmutableList.of(EXPECTED_URIS.get(0)));
    addExpectedPrefixObjects(PREFIXES.get(1), ImmutableList.of(EXPECTED_URIS.get(1)));
    EasyMock.expect(INPUT_DATA_CONFIG.getMaxListingLength()).andReturn(MAX_LISTING_LENGTH);
    EasyMock.replay(STORAGE);
    EasyMock.replay(INPUT_DATA_CONFIG);
    GoogleCloudStorageInputSource inputSource = new GoogleCloudStorageInputSource(STORAGE, INPUT_DATA_CONFIG, null, PREFIXES, null);
    Stream<InputSplit<List<CloudObjectLocation>>> splits = inputSource.createSplits(new JsonInputFormat(JSONPathSpec.DEFAULT, null, null), new MaxSizeSplitHintSpec(null, 1));
    Assert.assertEquals(EXPECTED_OBJECTS, splits.map(InputSplit::get).collect(Collectors.toList()));
}
Also used : JsonInputFormat(org.apache.druid.data.input.impl.JsonInputFormat) CloudObjectLocation(org.apache.druid.data.input.impl.CloudObjectLocation) InputSplit(org.apache.druid.data.input.InputSplit) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Aggregations

MaxSizeSplitHintSpec (org.apache.druid.data.input.MaxSizeSplitHintSpec)14 Test (org.junit.Test)11 InputSplit (org.apache.druid.data.input.InputSplit)10 CloudObjectLocation (org.apache.druid.data.input.impl.CloudObjectLocation)7 JsonInputFormat (org.apache.druid.data.input.impl.JsonInputFormat)6 HumanReadableBytes (org.apache.druid.java.util.common.HumanReadableBytes)6 InitializedNullHandlingTest (org.apache.druid.testing.InitializedNullHandlingTest)6 File (java.io.File)5 ImmutableList (com.google.common.collect.ImmutableList)2 URI (java.net.URI)2 List (java.util.List)2 CoordinatorCompactionConfig (org.apache.druid.server.coordinator.CoordinatorCompactionConfig)2 DataSourceCompactionConfig (org.apache.druid.server.coordinator.DataSourceCompactionConfig)2 UserCompactionTaskIOConfig (org.apache.druid.server.coordinator.UserCompactionTaskIOConfig)2 UserCompactionTaskQueryTuningConfig (org.apache.druid.server.coordinator.UserCompactionTaskQueryTuningConfig)2 JacksonInject (com.fasterxml.jackson.annotation.JacksonInject)1 JsonCreator (com.fasterxml.jackson.annotation.JsonCreator)1 JsonInclude (com.fasterxml.jackson.annotation.JsonInclude)1 JsonProperty (com.fasterxml.jackson.annotation.JsonProperty)1 Preconditions (com.google.common.base.Preconditions)1