Search in sources :

Example 1 with IndexColumn

use of com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn in project DataflowTemplates by GoogleCloudPlatform.

the class ChangeEventConvertor method changeEventToPrimaryKey.

static com.google.cloud.spanner.Key changeEventToPrimaryKey(Ddl ddl, JsonNode changeEvent) throws ChangeEventConvertorException {
    String tableName = changeEvent.get(DatastreamConstants.EVENT_TABLE_NAME_KEY).asText();
    try {
        Table table = ddl.table(tableName);
        ImmutableList<IndexColumn> keyColumns = table.primaryKeys();
        com.google.cloud.spanner.Key.Builder pk = com.google.cloud.spanner.Key.newBuilder();
        for (IndexColumn keyColumn : keyColumns) {
            Column key = table.column(keyColumn.name());
            Type keyColType = key.type();
            String keyColName = key.name().toLowerCase();
            switch(keyColType.getCode()) {
                case BOOL:
                    pk.append(ChangeEventTypeConvertor.toBoolean(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case INT64:
                    pk.append(ChangeEventTypeConvertor.toLong(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case FLOAT64:
                    pk.append(ChangeEventTypeConvertor.toDouble(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case STRING:
                    pk.append(ChangeEventTypeConvertor.toString(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case NUMERIC:
                    pk.append(ChangeEventTypeConvertor.toNumericBigDecimal(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case BYTES:
                    pk.append(ChangeEventTypeConvertor.toByteArray(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case TIMESTAMP:
                    pk.append(ChangeEventTypeConvertor.toTimestamp(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                case DATE:
                    pk.append(ChangeEventTypeConvertor.toDate(changeEvent, keyColName, /*requiredField=*/
                    true));
                    break;
                // TODO(b/179070999) -  Add support for other data types.
                default:
                    throw new IllegalArgumentException("Column name(" + keyColName + ") has unsupported column type(" + keyColType + ")");
            }
        }
        return pk.build();
    } catch (Exception e) {
        throw new ChangeEventConvertorException(e);
    }
}
Also used : Table(com.google.cloud.teleport.v2.templates.spanner.ddl.Table) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn) Type(com.google.cloud.spanner.Type) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn) Column(com.google.cloud.teleport.v2.templates.spanner.ddl.Column)

Example 2 with IndexColumn

use of com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn in project DataflowTemplates by GoogleCloudPlatform.

the class ChangeEventConvertor method changeEventToShadowTableMutationBuilder.

static Mutation.WriteBuilder changeEventToShadowTableMutationBuilder(Ddl ddl, JsonNode changeEvent, String shadowTablePrefix) throws ChangeEventConvertorException {
    String tableName = changeEvent.get(DatastreamConstants.EVENT_TABLE_NAME_KEY).asText();
    String shadowTableName = shadowTablePrefix + tableName;
    try {
        Table table = ddl.table(shadowTableName);
        ImmutableList<IndexColumn> keyColumns = table.primaryKeys();
        List<String> keyColumnNames = keyColumns.stream().map(IndexColumn::name).map(colName -> colName.toLowerCase()).collect(Collectors.toList());
        Set<String> requiredKeyColumnNames = new HashSet<>(keyColumnNames);
        Mutation.WriteBuilder builder = Mutation.newInsertOrUpdateBuilder(shadowTableName);
        populateMutationBuilderWithEvent(table, builder, changeEvent, keyColumnNames, requiredKeyColumnNames);
        return builder;
    } catch (Exception e) {
        throw new ChangeEventConvertorException(e);
    }
}
Also used : Iterator(java.util.Iterator) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) Spliterators(java.util.Spliterators) Set(java.util.Set) Type(com.google.cloud.spanner.Type) Mutation(com.google.cloud.spanner.Mutation) Value(com.google.cloud.spanner.Value) Collectors(java.util.stream.Collectors) ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) HashSet(java.util.HashSet) List(java.util.List) ImmutableList(com.google.common.collect.ImmutableList) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn) Column(com.google.cloud.teleport.v2.templates.spanner.ddl.Column) Table(com.google.cloud.teleport.v2.templates.spanner.ddl.Table) JsonNode(com.fasterxml.jackson.databind.JsonNode) StreamSupport(java.util.stream.StreamSupport) Spliterator(java.util.Spliterator) Table(com.google.cloud.teleport.v2.templates.spanner.ddl.Table) Mutation(com.google.cloud.spanner.Mutation) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn) HashSet(java.util.HashSet)

Example 3 with IndexColumn

use of com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn in project DataflowTemplates by GoogleCloudPlatform.

the class ShadowTableCreator method constructShadowTable.

/*
   * Constructs a shadow table for a data table in the information schema.
   * Note: Shadow tables for interleaved tables are not interleaved to
   * their shadow parent table.
   */
Table constructShadowTable(Ddl informationSchema, String dataTableName) {
    // Create a new shadow table with the given prefix.
    Table.Builder shadowTableBuilder = Table.builder();
    String shadowTableName = shadowTablePrefix + dataTableName;
    shadowTableBuilder.name(shadowTableName);
    // Add key columns from the data table to the shadow table builder.
    Table dataTable = informationSchema.table(dataTableName);
    Set<String> primaryKeyColNames = dataTable.primaryKeys().stream().map(k -> k.name()).collect(Collectors.toSet());
    List<Column> primaryKeyCols = dataTable.columns().stream().filter(col -> primaryKeyColNames.contains(col.name())).collect(Collectors.toList());
    for (Column col : primaryKeyCols) {
        shadowTableBuilder.addColumn(col);
    }
    // Add primary key constraints.
    for (IndexColumn keyColumn : dataTable.primaryKeys()) {
        if (keyColumn.order() == IndexColumn.Order.ASC) {
            shadowTableBuilder.primaryKey().asc(keyColumn.name()).end();
        } else if (keyColumn.order() == IndexColumn.Order.DESC) {
            shadowTableBuilder.primaryKey().desc(keyColumn.name()).end();
        }
    }
    // Add extra column to track ChangeEventSequence information
    addChangeEventSequenceColumns(shadowTableBuilder);
    return shadowTableBuilder.build();
}
Also used : List(java.util.List) Pair(org.apache.commons.lang3.tuple.Pair) DatastreamConstants(com.google.cloud.teleport.v2.templates.datastream.DatastreamConstants) Ddl(com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn) Column(com.google.cloud.teleport.v2.templates.spanner.ddl.Column) Map(java.util.Map) Table(com.google.cloud.teleport.v2.templates.spanner.ddl.Table) Set(java.util.Set) Collectors(java.util.stream.Collectors) Table(com.google.cloud.teleport.v2.templates.spanner.ddl.Table) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn) Column(com.google.cloud.teleport.v2.templates.spanner.ddl.Column) IndexColumn(com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn)

Aggregations

Column (com.google.cloud.teleport.v2.templates.spanner.ddl.Column)3 IndexColumn (com.google.cloud.teleport.v2.templates.spanner.ddl.IndexColumn)3 Table (com.google.cloud.teleport.v2.templates.spanner.ddl.Table)3 Type (com.google.cloud.spanner.Type)2 Ddl (com.google.cloud.teleport.v2.templates.spanner.ddl.Ddl)2 List (java.util.List)2 Set (java.util.Set)2 Collectors (java.util.stream.Collectors)2 JsonNode (com.fasterxml.jackson.databind.JsonNode)1 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)1 Mutation (com.google.cloud.spanner.Mutation)1 Value (com.google.cloud.spanner.Value)1 DatastreamConstants (com.google.cloud.teleport.v2.templates.datastream.DatastreamConstants)1 ImmutableList (com.google.common.collect.ImmutableList)1 HashSet (java.util.HashSet)1 Iterator (java.util.Iterator)1 Map (java.util.Map)1 Spliterator (java.util.Spliterator)1 Spliterators (java.util.Spliterators)1 StreamSupport (java.util.stream.StreamSupport)1