Search in sources :

Example 1 with InMemoryRepository

use of org.schemarepo.InMemoryRepository in project druid by druid-io.

the class AvroStreamInputFormatTest method testParse.

@Test
public void testParse() throws SchemaValidationException, IOException {
    Repository repository = new InMemoryRepository(null);
    AvroStreamInputFormat inputFormat = new AvroStreamInputFormat(flattenSpec, new SchemaRepoBasedAvroBytesDecoder<>(new Avro1124SubjectAndIdConverter(TOPIC), repository), false, false);
    NestedInputFormat inputFormat2 = jsonMapper.readValue(jsonMapper.writeValueAsString(inputFormat), NestedInputFormat.class);
    repository = ((SchemaRepoBasedAvroBytesDecoder) ((AvroStreamInputFormat) inputFormat2).getAvroBytesDecoder()).getSchemaRepository();
    // prepare data
    GenericRecord someAvroDatum = buildSomeAvroDatum();
    // encode schema id
    Avro1124SubjectAndIdConverter converter = new Avro1124SubjectAndIdConverter(TOPIC);
    TypedSchemaRepository<Integer, Schema, String> repositoryClient = new TypedSchemaRepository<>(repository, new IntegerConverter(), new AvroSchemaConverter(), new IdentityConverter());
    Integer id = repositoryClient.registerSchema(TOPIC, SomeAvroDatum.getClassSchema());
    ByteBuffer byteBuffer = ByteBuffer.allocate(4);
    converter.putSubjectAndId(id, byteBuffer);
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    out.write(byteBuffer.array());
    // encode data
    DatumWriter<GenericRecord> writer = new SpecificDatumWriter<>(someAvroDatum.getSchema());
    // write avro datum to bytes
    writer.write(someAvroDatum, EncoderFactory.get().directBinaryEncoder(out, null));
    final ByteEntity entity = new ByteEntity(ByteBuffer.wrap(out.toByteArray()));
    InputRow inputRow = inputFormat2.createReader(new InputRowSchema(timestampSpec, dimensionsSpec, null), entity, null).read().next();
    assertInputRowCorrect(inputRow, DIMENSIONS, false);
}
Also used : Avro1124SubjectAndIdConverter(org.apache.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter) NestedInputFormat(org.apache.druid.data.input.impl.NestedInputFormat) AvroSchemaConverter(org.schemarepo.api.converter.AvroSchemaConverter) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) ByteEntity(org.apache.druid.data.input.impl.ByteEntity) Schema(org.apache.avro.Schema) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ByteBuffer(java.nio.ByteBuffer) SpecificDatumWriter(org.apache.avro.specific.SpecificDatumWriter) IntegerConverter(org.schemarepo.api.converter.IntegerConverter) Repository(org.schemarepo.Repository) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) IdentityConverter(org.schemarepo.api.converter.IdentityConverter) GenericRecord(org.apache.avro.generic.GenericRecord) AvroStreamInputFormat(org.apache.druid.data.input.avro.AvroStreamInputFormat) Test(org.junit.Test)

Example 2 with InMemoryRepository

use of org.schemarepo.InMemoryRepository in project druid by druid-io.

the class AvroStreamInputRowParserTest method testParse.

@Test
public void testParse() throws SchemaValidationException, IOException {
    // serde test
    Repository repository = new InMemoryRepository(null);
    AvroStreamInputRowParser parser = new AvroStreamInputRowParser(PARSE_SPEC, new SchemaRepoBasedAvroBytesDecoder<String, Integer>(new Avro1124SubjectAndIdConverter(TOPIC), repository));
    ByteBufferInputRowParser parser2 = jsonMapper.readValue(jsonMapper.writeValueAsString(parser), ByteBufferInputRowParser.class);
    repository = ((SchemaRepoBasedAvroBytesDecoder) ((AvroStreamInputRowParser) parser2).getAvroBytesDecoder()).getSchemaRepository();
    // prepare data
    GenericRecord someAvroDatum = buildSomeAvroDatum();
    // encode schema id
    Avro1124SubjectAndIdConverter converter = new Avro1124SubjectAndIdConverter(TOPIC);
    TypedSchemaRepository<Integer, Schema, String> repositoryClient = new TypedSchemaRepository<Integer, Schema, String>(repository, new IntegerConverter(), new AvroSchemaConverter(), new IdentityConverter());
    Integer id = repositoryClient.registerSchema(TOPIC, SomeAvroDatum.getClassSchema());
    ByteBuffer byteBuffer = ByteBuffer.allocate(4);
    converter.putSubjectAndId(TOPIC, id, byteBuffer);
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    out.write(byteBuffer.array());
    // encode data
    DatumWriter<GenericRecord> writer = new GenericDatumWriter<GenericRecord>(someAvroDatum.getSchema());
    // write avro datum to bytes
    writer.write(someAvroDatum, EncoderFactory.get().directBinaryEncoder(out, null));
    InputRow inputRow = parser2.parse(ByteBuffer.wrap(out.toByteArray()));
    assertInputRowCorrect(inputRow);
}
Also used : Avro1124SubjectAndIdConverter(io.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter) AvroSchemaConverter(org.schemarepo.api.converter.AvroSchemaConverter) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) Schema(org.apache.avro.Schema) ByteArrayOutputStream(java.io.ByteArrayOutputStream) GenericDatumWriter(org.apache.avro.generic.GenericDatumWriter) ByteBuffer(java.nio.ByteBuffer) IntegerConverter(org.schemarepo.api.converter.IntegerConverter) Repository(org.schemarepo.Repository) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) IdentityConverter(org.schemarepo.api.converter.IdentityConverter) GenericRecord(org.apache.avro.generic.GenericRecord) Test(org.junit.Test)

Example 3 with InMemoryRepository

use of org.schemarepo.InMemoryRepository in project druid by druid-io.

the class AvroStreamInputRowParserTest method testParseSchemaless.

@Test
public void testParseSchemaless() throws SchemaValidationException, IOException {
    // serde test
    Repository repository = new InMemoryRepository(null);
    AvroStreamInputRowParser parser = new AvroStreamInputRowParser(PARSE_SPEC_SCHEMALESS, new SchemaRepoBasedAvroBytesDecoder<>(new Avro1124SubjectAndIdConverter(TOPIC), repository), false, false);
    ByteBufferInputRowParser parser2 = jsonMapper.readValue(jsonMapper.writeValueAsString(parser), ByteBufferInputRowParser.class);
    repository = ((SchemaRepoBasedAvroBytesDecoder) ((AvroStreamInputRowParser) parser2).getAvroBytesDecoder()).getSchemaRepository();
    // prepare data
    GenericRecord someAvroDatum = buildSomeAvroDatum();
    // encode schema id
    Avro1124SubjectAndIdConverter converter = new Avro1124SubjectAndIdConverter(TOPIC);
    TypedSchemaRepository<Integer, Schema, String> repositoryClient = new TypedSchemaRepository<>(repository, new IntegerConverter(), new AvroSchemaConverter(), new IdentityConverter());
    Integer id = repositoryClient.registerSchema(TOPIC, SomeAvroDatum.getClassSchema());
    ByteBuffer byteBuffer = ByteBuffer.allocate(4);
    converter.putSubjectAndId(id, byteBuffer);
    try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
        out.write(byteBuffer.array());
        // encode data
        DatumWriter<GenericRecord> writer = new SpecificDatumWriter<>(someAvroDatum.getSchema());
        // write avro datum to bytes
        writer.write(someAvroDatum, EncoderFactory.get().directBinaryEncoder(out, null));
        InputRow inputRow = parser2.parseBatch(ByteBuffer.wrap(out.toByteArray())).get(0);
        assertInputRowCorrect(inputRow, DIMENSIONS_SCHEMALESS, false);
    }
}
Also used : Avro1124SubjectAndIdConverter(org.apache.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter) AvroSchemaConverter(org.schemarepo.api.converter.AvroSchemaConverter) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) Schema(org.apache.avro.Schema) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ByteBuffer(java.nio.ByteBuffer) SpecificDatumWriter(org.apache.avro.specific.SpecificDatumWriter) IntegerConverter(org.schemarepo.api.converter.IntegerConverter) Repository(org.schemarepo.Repository) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) IdentityConverter(org.schemarepo.api.converter.IdentityConverter) GenericRecord(org.apache.avro.generic.GenericRecord) Test(org.junit.Test)

Example 4 with InMemoryRepository

use of org.schemarepo.InMemoryRepository in project druid by druid-io.

the class AvroStreamInputRowParserTest method testParse.

@Test
public void testParse() throws SchemaValidationException, IOException {
    // serde test
    Repository repository = new InMemoryRepository(null);
    AvroStreamInputRowParser parser = new AvroStreamInputRowParser(PARSE_SPEC, new SchemaRepoBasedAvroBytesDecoder<>(new Avro1124SubjectAndIdConverter(TOPIC), repository), false, false);
    ByteBufferInputRowParser parser2 = jsonMapper.readValue(jsonMapper.writeValueAsString(parser), ByteBufferInputRowParser.class);
    repository = ((SchemaRepoBasedAvroBytesDecoder) ((AvroStreamInputRowParser) parser2).getAvroBytesDecoder()).getSchemaRepository();
    // prepare data
    GenericRecord someAvroDatum = buildSomeAvroDatum();
    // encode schema id
    Avro1124SubjectAndIdConverter converter = new Avro1124SubjectAndIdConverter(TOPIC);
    TypedSchemaRepository<Integer, Schema, String> repositoryClient = new TypedSchemaRepository<>(repository, new IntegerConverter(), new AvroSchemaConverter(), new IdentityConverter());
    Integer id = repositoryClient.registerSchema(TOPIC, SomeAvroDatum.getClassSchema());
    ByteBuffer byteBuffer = ByteBuffer.allocate(4);
    converter.putSubjectAndId(id, byteBuffer);
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    out.write(byteBuffer.array());
    // encode data
    DatumWriter<GenericRecord> writer = new SpecificDatumWriter<>(someAvroDatum.getSchema());
    // write avro datum to bytes
    writer.write(someAvroDatum, EncoderFactory.get().directBinaryEncoder(out, null));
    InputRow inputRow = parser2.parseBatch(ByteBuffer.wrap(out.toByteArray())).get(0);
    assertInputRowCorrect(inputRow, DIMENSIONS, false);
}
Also used : Avro1124SubjectAndIdConverter(org.apache.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter) AvroSchemaConverter(org.schemarepo.api.converter.AvroSchemaConverter) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) Schema(org.apache.avro.Schema) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ByteBuffer(java.nio.ByteBuffer) SpecificDatumWriter(org.apache.avro.specific.SpecificDatumWriter) IntegerConverter(org.schemarepo.api.converter.IntegerConverter) Repository(org.schemarepo.Repository) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) IdentityConverter(org.schemarepo.api.converter.IdentityConverter) GenericRecord(org.apache.avro.generic.GenericRecord) Test(org.junit.Test)

Example 5 with InMemoryRepository

use of org.schemarepo.InMemoryRepository in project druid by druid-io.

the class AvroStreamInputFormatTest method testParseSchemaless.

@Test
public void testParseSchemaless() throws SchemaValidationException, IOException {
    Repository repository = new InMemoryRepository(null);
    AvroStreamInputFormat inputFormat = new AvroStreamInputFormat(flattenSpec, new SchemaRepoBasedAvroBytesDecoder<>(new Avro1124SubjectAndIdConverter(TOPIC), repository), false, false);
    NestedInputFormat inputFormat2 = jsonMapper.readValue(jsonMapper.writeValueAsString(inputFormat), NestedInputFormat.class);
    repository = ((SchemaRepoBasedAvroBytesDecoder) ((AvroStreamInputFormat) inputFormat2).getAvroBytesDecoder()).getSchemaRepository();
    // prepare data
    GenericRecord someAvroDatum = buildSomeAvroDatum();
    // encode schema id
    Avro1124SubjectAndIdConverter converter = new Avro1124SubjectAndIdConverter(TOPIC);
    TypedSchemaRepository<Integer, Schema, String> repositoryClient = new TypedSchemaRepository<>(repository, new IntegerConverter(), new AvroSchemaConverter(), new IdentityConverter());
    Integer id = repositoryClient.registerSchema(TOPIC, SomeAvroDatum.getClassSchema());
    ByteBuffer byteBuffer = ByteBuffer.allocate(4);
    converter.putSubjectAndId(id, byteBuffer);
    try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
        out.write(byteBuffer.array());
        // encode data
        DatumWriter<GenericRecord> writer = new SpecificDatumWriter<>(someAvroDatum.getSchema());
        // write avro datum to bytes
        writer.write(someAvroDatum, EncoderFactory.get().directBinaryEncoder(out, null));
        final ByteEntity entity = new ByteEntity(ByteBuffer.wrap(out.toByteArray()));
        InputRow inputRow = inputFormat2.createReader(new InputRowSchema(timestampSpec, DimensionsSpec.EMPTY, null), entity, null).read().next();
        assertInputRowCorrect(inputRow, DIMENSIONS_SCHEMALESS, false);
    }
}
Also used : Avro1124SubjectAndIdConverter(org.apache.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter) NestedInputFormat(org.apache.druid.data.input.impl.NestedInputFormat) AvroSchemaConverter(org.schemarepo.api.converter.AvroSchemaConverter) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) ByteEntity(org.apache.druid.data.input.impl.ByteEntity) Schema(org.apache.avro.Schema) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ByteBuffer(java.nio.ByteBuffer) SpecificDatumWriter(org.apache.avro.specific.SpecificDatumWriter) IntegerConverter(org.schemarepo.api.converter.IntegerConverter) Repository(org.schemarepo.Repository) InMemoryRepository(org.schemarepo.InMemoryRepository) TypedSchemaRepository(org.schemarepo.api.TypedSchemaRepository) IdentityConverter(org.schemarepo.api.converter.IdentityConverter) GenericRecord(org.apache.avro.generic.GenericRecord) AvroStreamInputFormat(org.apache.druid.data.input.avro.AvroStreamInputFormat) Test(org.junit.Test)

Aggregations

ByteArrayOutputStream (java.io.ByteArrayOutputStream)5 ByteBuffer (java.nio.ByteBuffer)5 Schema (org.apache.avro.Schema)5 GenericRecord (org.apache.avro.generic.GenericRecord)5 Test (org.junit.Test)5 InMemoryRepository (org.schemarepo.InMemoryRepository)5 Repository (org.schemarepo.Repository)5 TypedSchemaRepository (org.schemarepo.api.TypedSchemaRepository)5 AvroSchemaConverter (org.schemarepo.api.converter.AvroSchemaConverter)5 IdentityConverter (org.schemarepo.api.converter.IdentityConverter)5 IntegerConverter (org.schemarepo.api.converter.IntegerConverter)5 SpecificDatumWriter (org.apache.avro.specific.SpecificDatumWriter)4 Avro1124SubjectAndIdConverter (org.apache.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter)4 AvroStreamInputFormat (org.apache.druid.data.input.avro.AvroStreamInputFormat)2 ByteEntity (org.apache.druid.data.input.impl.ByteEntity)2 NestedInputFormat (org.apache.druid.data.input.impl.NestedInputFormat)2 Avro1124SubjectAndIdConverter (io.druid.data.input.schemarepo.Avro1124SubjectAndIdConverter)1 GenericDatumWriter (org.apache.avro.generic.GenericDatumWriter)1