Search in sources :

Example 16 with HumanReadableBytes

use of org.apache.druid.java.util.common.HumanReadableBytes in project druid by druid-io.

the class OssInputSourceTest method testCreateSplitsWithSplitHintSpecRespectingHint.

@Test
public void testCreateSplitsWithSplitHintSpecRespectingHint() {
    EasyMock.reset(OSSCLIENT);
    expectListObjects(PREFIXES.get(0), ImmutableList.of(EXPECTED_URIS.get(0)), CONTENT);
    expectListObjects(PREFIXES.get(1), ImmutableList.of(EXPECTED_URIS.get(1)), CONTENT);
    EasyMock.replay(OSSCLIENT);
    OssInputSource inputSource = new OssInputSource(OSSCLIENT, INPUT_DATA_CONFIG, null, PREFIXES, null, null);
    Stream<InputSplit<List<CloudObjectLocation>>> splits = inputSource.createSplits(new JsonInputFormat(JSONPathSpec.DEFAULT, null, null), new MaxSizeSplitHintSpec(new HumanReadableBytes(CONTENT.length * 3L), null));
    Assert.assertEquals(ImmutableList.of(EXPECTED_URIS.stream().map(CloudObjectLocation::new).collect(Collectors.toList())), splits.map(InputSplit::get).collect(Collectors.toList()));
    EasyMock.verify(OSSCLIENT);
}
Also used : JsonInputFormat(org.apache.druid.data.input.impl.JsonInputFormat) CloudObjectLocation(org.apache.druid.data.input.impl.CloudObjectLocation) HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) InputSplit(org.apache.druid.data.input.InputSplit) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Example 17 with HumanReadableBytes

use of org.apache.druid.java.util.common.HumanReadableBytes in project druid by druid-io.

the class GoogleCloudStorageInputSourceTest method testCreateSplitsWithSplitHintSpecRespectingHint.

@Test
public void testCreateSplitsWithSplitHintSpecRespectingHint() throws IOException {
    EasyMock.reset(STORAGE);
    EasyMock.reset(INPUT_DATA_CONFIG);
    addExpectedPrefixObjects(PREFIXES.get(0), ImmutableList.of(EXPECTED_URIS.get(0)));
    addExpectedPrefixObjects(PREFIXES.get(1), ImmutableList.of(EXPECTED_URIS.get(1)));
    EasyMock.expect(INPUT_DATA_CONFIG.getMaxListingLength()).andReturn(MAX_LISTING_LENGTH);
    EasyMock.replay(STORAGE);
    EasyMock.replay(INPUT_DATA_CONFIG);
    GoogleCloudStorageInputSource inputSource = new GoogleCloudStorageInputSource(STORAGE, INPUT_DATA_CONFIG, null, PREFIXES, null);
    Stream<InputSplit<List<CloudObjectLocation>>> splits = inputSource.createSplits(new JsonInputFormat(JSONPathSpec.DEFAULT, null, null), new MaxSizeSplitHintSpec(new HumanReadableBytes(CONTENT.length * 3L), null));
    Assert.assertEquals(ImmutableList.of(EXPECTED_URIS.stream().map(CloudObjectLocation::new).collect(Collectors.toList())), splits.map(InputSplit::get).collect(Collectors.toList()));
}
Also used : JsonInputFormat(org.apache.druid.data.input.impl.JsonInputFormat) CloudObjectLocation(org.apache.druid.data.input.impl.CloudObjectLocation) HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) InputSplit(org.apache.druid.data.input.InputSplit) MaxSizeSplitHintSpec(org.apache.druid.data.input.MaxSizeSplitHintSpec) InitializedNullHandlingTest(org.apache.druid.testing.InitializedNullHandlingTest) Test(org.junit.Test)

Example 18 with HumanReadableBytes

use of org.apache.druid.java.util.common.HumanReadableBytes in project druid by druid-io.

the class MaxSizeSplitHintSpecTest method testSplitSmallInputsWithMaxNumFilesEachSplitShouldHaveLessFilesAssigned.

@Test
public void testSplitSmallInputsWithMaxNumFilesEachSplitShouldHaveLessFilesAssigned() {
    final int eachInputSize = 3;
    final MaxSizeSplitHintSpec splitHintSpec = new MaxSizeSplitHintSpec(new HumanReadableBytes("500M"), 3);
    final Function<Integer, InputFileAttribute> inputAttributeExtractor = InputFileAttribute::new;
    final List<List<Integer>> splits = Lists.newArrayList(splitHintSpec.split(IntStream.generate(() -> eachInputSize).limit(10).iterator(), inputAttributeExtractor));
    Assert.assertEquals(4, splits.size());
    Assert.assertEquals(3, splits.get(0).size());
    Assert.assertEquals(3, splits.get(1).size());
    Assert.assertEquals(3, splits.get(2).size());
    Assert.assertEquals(1, splits.get(3).size());
}
Also used : List(java.util.List) HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) Test(org.junit.Test)

Example 19 with HumanReadableBytes

use of org.apache.druid.java.util.common.HumanReadableBytes in project druid by druid-io.

the class MaxSizeSplitHintSpecTest method testSerde.

@Test
public void testSerde() throws IOException {
    final ObjectMapper mapper = new ObjectMapper();
    final MaxSizeSplitHintSpec original = new MaxSizeSplitHintSpec(new HumanReadableBytes(1024L), 20_000);
    final byte[] bytes = mapper.writeValueAsBytes(original);
    final MaxSizeSplitHintSpec fromJson = (MaxSizeSplitHintSpec) mapper.readValue(bytes, SplitHintSpec.class);
    Assert.assertEquals(original, fromJson);
}
Also used : HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Test(org.junit.Test)

Example 20 with HumanReadableBytes

use of org.apache.druid.java.util.common.HumanReadableBytes in project druid by druid-io.

the class MaxSizeSplitHintSpecTest method testReadFromJson.

@Test
public void testReadFromJson() throws JsonProcessingException {
    final ObjectMapper mapper = new ObjectMapper();
    final String json = "{" + "  \"type\":\"maxSize\"," + "  \"maxSplitSize\":1024," + "  \"maxNumFiles\":20000" + "}\n";
    final MaxSizeSplitHintSpec fromJson = (MaxSizeSplitHintSpec) mapper.readValue(json, SplitHintSpec.class);
    Assert.assertEquals(new MaxSizeSplitHintSpec(new HumanReadableBytes(1024L), 20_000), fromJson);
}
Also used : HumanReadableBytes(org.apache.druid.java.util.common.HumanReadableBytes) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Test(org.junit.Test)

Aggregations

HumanReadableBytes (org.apache.druid.java.util.common.HumanReadableBytes)39 Test (org.junit.Test)35 InitializedNullHandlingTest (org.apache.druid.testing.InitializedNullHandlingTest)19 List (java.util.List)7 MaxSizeSplitHintSpec (org.apache.druid.data.input.MaxSizeSplitHintSpec)6 ExpressionLambdaAggregatorFactory (org.apache.druid.query.aggregation.ExpressionLambdaAggregatorFactory)6 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)4 InputSplit (org.apache.druid.data.input.InputSplit)4 SegmentsSplitHintSpec (org.apache.druid.data.input.SegmentsSplitHintSpec)4 DynamicPartitionsSpec (org.apache.druid.indexer.partitions.DynamicPartitionsSpec)4 ImmutableSet (com.google.common.collect.ImmutableSet)3 File (java.io.File)3 Collectors (java.util.stream.Collectors)3 Nullable (javax.annotation.Nullable)3 AggregateCall (org.apache.calcite.rel.core.AggregateCall)3 Project (org.apache.calcite.rel.core.Project)3 RexBuilder (org.apache.calcite.rex.RexBuilder)3 RexLiteral (org.apache.calcite.rex.RexLiteral)3 RexNode (org.apache.calcite.rex.RexNode)3 SqlAggFunction (org.apache.calcite.sql.SqlAggFunction)3