Search in sources :

Example 1 with DecoderColumnHandle

use of io.trino.decoder.DecoderColumnHandle in project trino by trinodb.

the class TestAvroDecoder method testSchemaEvolutionRemovingColumn.

@Test
public void testSchemaEvolutionRemovingColumn() {
    byte[] originalData = buildAvroData(getFieldBuilder().name("string_field").type().stringType().noDefault().name("string_field_to_be_removed").type().optional().stringType().endRecord(), ImmutableMap.of("string_field", "string_field_value", "string_field_to_be_removed", "removed_field_value"));
    DecoderTestColumnHandle evolvedColumn = new DecoderTestColumnHandle(0, "row0", VARCHAR, "string_field", null, null, false, false, false);
    String removedColumnSchema = getFieldBuilder().name("string_field").type().stringType().noDefault().endRecord().toString();
    Map<DecoderColumnHandle, FieldValueProvider> decodedEvolvedRow = decodeRow(originalData, ImmutableSet.of(evolvedColumn), ImmutableMap.of(DATA_SCHEMA, removedColumnSchema));
    assertEquals(decodedEvolvedRow.size(), 1);
    checkValue(decodedEvolvedRow, evolvedColumn, "string_field_value");
}
Also used : DecoderTestColumnHandle(io.trino.decoder.DecoderTestColumnHandle) FieldValueProvider(io.trino.decoder.FieldValueProvider) DecoderColumnHandle(io.trino.decoder.DecoderColumnHandle) Test(org.testng.annotations.Test)

Example 2 with DecoderColumnHandle

use of io.trino.decoder.DecoderColumnHandle in project trino by trinodb.

the class TestAvroDecoder method testSchemaEvolutionIntToDouble.

@Test
public void testSchemaEvolutionIntToDouble() {
    byte[] originalIntData = buildAvroData(getFieldBuilder().name("int_to_double_field").type().intType().noDefault().endRecord(), "int_to_double_field", 100);
    DecoderTestColumnHandle doubleColumnReadingIntData = new DecoderTestColumnHandle(0, "row0", DOUBLE, "int_to_double_field", null, null, false, false, false);
    String changedTypeSchema = getFieldBuilder().name("int_to_double_field").type().doubleType().noDefault().endRecord().toString();
    Map<DecoderColumnHandle, FieldValueProvider> decodedEvolvedRow = decodeRow(originalIntData, ImmutableSet.of(doubleColumnReadingIntData), ImmutableMap.of(DATA_SCHEMA, changedTypeSchema));
    assertEquals(decodedEvolvedRow.size(), 1);
    checkValue(decodedEvolvedRow, doubleColumnReadingIntData, 100.0);
}
Also used : DecoderTestColumnHandle(io.trino.decoder.DecoderTestColumnHandle) FieldValueProvider(io.trino.decoder.FieldValueProvider) DecoderColumnHandle(io.trino.decoder.DecoderColumnHandle) Test(org.testng.annotations.Test)

Example 3 with DecoderColumnHandle

use of io.trino.decoder.DecoderColumnHandle in project trino by trinodb.

the class TestAvroDecoder method testBytesDecodedAsVarbinary.

@Test
public void testBytesDecodedAsVarbinary() {
    DecoderTestColumnHandle row = new DecoderTestColumnHandle(0, "row", VARBINARY, "encoded", null, null, false, false, false);
    Map<DecoderColumnHandle, FieldValueProvider> decodedRow = buildAndDecodeColumn(row, "encoded", "\"bytes\"", ByteBuffer.wrap("mytext".getBytes(UTF_8)));
    checkValue(decodedRow, row, "mytext");
}
Also used : DecoderTestColumnHandle(io.trino.decoder.DecoderTestColumnHandle) FieldValueProvider(io.trino.decoder.FieldValueProvider) DecoderColumnHandle(io.trino.decoder.DecoderColumnHandle) Test(org.testng.annotations.Test)

Example 4 with DecoderColumnHandle

use of io.trino.decoder.DecoderColumnHandle in project trino by trinodb.

the class TestAvroDecoder method testMapDecodedAsMap.

@Test
public void testMapDecodedAsMap() {
    DecoderTestColumnHandle row = new DecoderTestColumnHandle(0, "row", VARCHAR_MAP_TYPE, "map_field", null, null, false, false, false);
    Map<DecoderColumnHandle, FieldValueProvider> decodedRow = buildAndDecodeColumn(row, "map_field", "{\"type\": \"map\", \"values\": \"string\"}", ImmutableMap.of("key1", "abc", "key2", "def", "key3", "zyx"));
    checkMapValue(decodedRow, row, ImmutableMap.of("key1", "abc", "key2", "def", "key3", "zyx"));
}
Also used : DecoderTestColumnHandle(io.trino.decoder.DecoderTestColumnHandle) FieldValueProvider(io.trino.decoder.FieldValueProvider) DecoderColumnHandle(io.trino.decoder.DecoderColumnHandle) Test(org.testng.annotations.Test)

Example 5 with DecoderColumnHandle

use of io.trino.decoder.DecoderColumnHandle in project trino by trinodb.

the class TestAvroDecoder method testSchemaEvolutionAddingColumn.

@Test
public void testSchemaEvolutionAddingColumn() {
    DecoderTestColumnHandle originalColumn = new DecoderTestColumnHandle(0, "row0", VARCHAR, "string_field", null, null, false, false, false);
    DecoderTestColumnHandle newlyAddedColumn = new DecoderTestColumnHandle(1, "row1", VARCHAR, "string_field_added", null, null, false, false, false);
    // the decoded avro data file does not have string_field_added
    byte[] originalData = buildAvroData(getFieldBuilder().name("string_field").type().stringType().noDefault().endRecord(), "string_field", "string_field_value");
    String addedColumnSchema = getFieldBuilder().name("string_field").type().stringType().noDefault().name("string_field_added").type().optional().stringType().endRecord().toString();
    Map<DecoderColumnHandle, FieldValueProvider> decodedRow = decodeRow(originalData, ImmutableSet.of(originalColumn, newlyAddedColumn), ImmutableMap.of(DATA_SCHEMA, addedColumnSchema));
    assertEquals(decodedRow.size(), 2);
    checkValue(decodedRow, originalColumn, "string_field_value");
    checkIsNull(decodedRow, newlyAddedColumn);
}
Also used : DecoderTestColumnHandle(io.trino.decoder.DecoderTestColumnHandle) FieldValueProvider(io.trino.decoder.FieldValueProvider) DecoderColumnHandle(io.trino.decoder.DecoderColumnHandle) Test(org.testng.annotations.Test)

Aggregations

DecoderColumnHandle (io.trino.decoder.DecoderColumnHandle)66 FieldValueProvider (io.trino.decoder.FieldValueProvider)63 DecoderTestColumnHandle (io.trino.decoder.DecoderTestColumnHandle)61 Test (org.testng.annotations.Test)61 Schema (org.apache.avro.Schema)24 RowDecoder (io.trino.decoder.RowDecoder)17 ArrayType (io.trino.spi.type.ArrayType)15 ImmutableList (com.google.common.collect.ImmutableList)13 ArrayList (java.util.ArrayList)13 List (java.util.List)12 GenericArray (org.apache.avro.generic.GenericArray)11 Map (java.util.Map)9 HashMap (java.util.HashMap)8 ImmutableMap (com.google.common.collect.ImmutableMap)6 ByteBuffer (java.nio.ByteBuffer)4 GenericData (org.apache.avro.generic.GenericData)4 Collections.emptyMap (java.util.Collections.emptyMap)3 GenericRecord (org.apache.avro.generic.GenericRecord)3 GenericRecordBuilder (org.apache.avro.generic.GenericRecordBuilder)3 MockSchemaRegistryClient (io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient)2