Search in sources :

Example 1 with AvroToEntity

use of com.google.cloud.teleport.templates.common.BigQueryConverters.AvroToEntity in project DataflowTemplates by GoogleCloudPlatform.

the class BigQueryConvertersTest method testAvroToEntityDefaultNamespace.

/**
 * Tests that {@link BigQueryConverters.AvroToEntity} creates an Entity with a default namespace
 * when the namespace is not specified.
 */
@Test
public void testAvroToEntityDefaultNamespace() throws Exception {
    // Create test data
    List<TableFieldSchema> fields = new ArrayList<>();
    fields.add(new TableFieldSchema().setName(idField).setType("STRING"));
    fields.add(new TableFieldSchema().setName(shortStringField).setType("STRING"));
    TableSchema bqSchema = new TableSchema().setFields(fields);
    Schema avroSchema = new Schema.Parser().parse(String.format(avroSchemaTemplate, new StringBuilder().append(String.format(avroFieldTemplate, idField, "int", idFieldDesc)).append(",").append(generateShortStringField()).toString()));
    GenericRecordBuilder builder = new GenericRecordBuilder(avroSchema);
    builder.set(idField, 1);
    builder.set(shortStringField, shortStringFieldValue);
    Record record = builder.build();
    SchemaAndRecord inputBqData = new SchemaAndRecord(record, bqSchema);
    // Run the test
    AvroToEntity noNamespaceConverter = AvroToEntity.newBuilder().setEntityKind(entityKind).setUniqueNameColumn(uniqueNameColumn).build();
    Entity outputEntity = noNamespaceConverter.apply(inputBqData);
    // Assess results
    assertTrue(outputEntity.hasKey());
    assertEquals("", outputEntity.getKey().getPartitionId().getNamespaceId());
}
Also used : Entity(com.google.datastore.v1.Entity) AvroToEntity(com.google.cloud.teleport.templates.common.BigQueryConverters.AvroToEntity) TableSchema(com.google.api.services.bigquery.model.TableSchema) TableSchema(com.google.api.services.bigquery.model.TableSchema) Schema(org.apache.avro.Schema) TableFieldSchema(com.google.api.services.bigquery.model.TableFieldSchema) ArrayList(java.util.ArrayList) GenericRecordBuilder(org.apache.avro.generic.GenericRecordBuilder) Record(org.apache.avro.generic.GenericData.Record) SchemaAndRecord(org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord) TableFieldSchema(com.google.api.services.bigquery.model.TableFieldSchema) SchemaAndRecord(org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord) AvroToEntity(com.google.cloud.teleport.templates.common.BigQueryConverters.AvroToEntity) Test(org.junit.Test)

Aggregations

TableFieldSchema (com.google.api.services.bigquery.model.TableFieldSchema)1 TableSchema (com.google.api.services.bigquery.model.TableSchema)1 AvroToEntity (com.google.cloud.teleport.templates.common.BigQueryConverters.AvroToEntity)1 Entity (com.google.datastore.v1.Entity)1 ArrayList (java.util.ArrayList)1 Schema (org.apache.avro.Schema)1 Record (org.apache.avro.generic.GenericData.Record)1 GenericRecordBuilder (org.apache.avro.generic.GenericRecordBuilder)1 SchemaAndRecord (org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord)1 Test (org.junit.Test)1