Examples with OnHeapMemorySegmentWriteOutMedium org.apache.druid.segment.writeout.OnHeapMemorySegmentWriteOutMedium used on opensource projects

Example 1 with OnHeapMemorySegmentWriteOutMedium

use of org.apache.druid.segment.writeout.OnHeapMemorySegmentWriteOutMedium in project druid by druid-io.

the class GenericIndexedWriterTest method writeLargeValueIntoLargeColumn.

@Test
public void writeLargeValueIntoLargeColumn() throws IOException {
    // Regression test for https://github.com/apache/druid/issues/9027.
    final GenericIndexedWriter<String> writer = new GenericIndexedWriter<>(new OnHeapMemorySegmentWriteOutMedium(), "test", GenericIndexed.STRING_STRATEGY, 100);
    writer.setIntMaxForCasting(150);
    writer.open();
    writer.write("i really like writing strings");
    writer.write("i really like writing strings");
    writer.write("i really like writing strings i really like writing strings i really like writing strings");
    writer.write("i really like writing strings");
    writer.writeTo(FileChannel.open(temporaryFolder.newFile().toPath(), StandardOpenOption.WRITE), new FileSmoosher(temporaryFolder.newFolder()));
}

Example 2 with OnHeapMemorySegmentWriteOutMedium

use of org.apache.druid.segment.writeout.OnHeapMemorySegmentWriteOutMedium in project druid by druid-io.

the class BaseColumnarLongsBenchmark method encodeToFile.

static int encodeToFile(long[] vals, String encoding, FileChannel output) throws IOException {
    SegmentWriteOutMedium writeOutMedium = new OnHeapMemorySegmentWriteOutMedium();
    ColumnarLongsSerializer serializer;
    switch(encoding) {
        case "lz4-longs":
            serializer = CompressionFactory.getLongSerializer(encoding, writeOutMedium, "lz4-longs", ByteOrder.LITTLE_ENDIAN, CompressionFactory.LongEncodingStrategy.LONGS, CompressionStrategy.LZ4);
            break;
        case "lz4-auto":
            serializer = CompressionFactory.getLongSerializer(encoding, writeOutMedium, "lz4-auto", ByteOrder.LITTLE_ENDIAN, CompressionFactory.LongEncodingStrategy.AUTO, CompressionStrategy.LZ4);
            break;
        case "none-longs":
            serializer = CompressionFactory.getLongSerializer(encoding, writeOutMedium, "none-longs", ByteOrder.LITTLE_ENDIAN, CompressionFactory.LongEncodingStrategy.LONGS, CompressionStrategy.NONE);
            break;
        case "none-auto":
            serializer = CompressionFactory.getLongSerializer(encoding, writeOutMedium, "none-auto", ByteOrder.LITTLE_ENDIAN, CompressionFactory.LongEncodingStrategy.AUTO, CompressionStrategy.NONE);
            break;
        default:
            throw new RuntimeException("unknown encoding");
    }
    serializer.open();
    for (long val : vals) {
        serializer.add(val);
    }
    serializer.writeTo(output, null);
    return (int) serializer.getSerializedSize();
}

Example 3 with OnHeapMemorySegmentWriteOutMedium

use of org.apache.druid.segment.writeout.OnHeapMemorySegmentWriteOutMedium in project druid by druid-io.

the class GenericIndexedStringWriterTest method testRandomAccess.

@Test
public void testRandomAccess() throws IOException {
    OnHeapMemorySegmentWriteOutMedium segmentWriteOutMedium = new OnHeapMemorySegmentWriteOutMedium();
    GenericIndexedWriter<String> writer = new GenericIndexedWriter<>(segmentWriteOutMedium, "test", GenericIndexed.STRING_STRATEGY);
    writer.open();
    writer.write(null);
    List<String> strings = new ArrayList<>();
    strings.add(null);
    ThreadLocalRandom r = ThreadLocalRandom.current();
    for (int i = 0; i < 100_000; i++) {
        byte[] bs = new byte[r.nextInt(1, 10)];
        r.nextBytes(bs);
        String s = new String(bs, StandardCharsets.US_ASCII);
        strings.add(s);
        writer.write(s);
    }
    for (int i = 0; i < strings.size(); i++) {
        Assert.assertEquals(strings.get(i), writer.get(i));
    }
}