Search in sources :

Example 1 with MysqlMutationMetadata

use of com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata in project SpinalTap by airbnb.

the class AbstractMysqlSource method commitCheckpoint.

public void commitCheckpoint(Mutation<?> mutation) {
    SourceState savedState = lastSavedState.get();
    if (mutation == null || savedState == null) {
        return;
    }
    Preconditions.checkState(mutation instanceof MysqlMutation);
    MysqlMutationMetadata metadata = ((MysqlMutation) mutation).getMetadata();
    // Make sure we are saving at a higher watermark
    if (savedState.getLastOffset() >= metadata.getId()) {
        return;
    }
    SourceState newState = new SourceState(metadata.getTimestamp(), metadata.getId(), currentLeaderEpoch.get(), metadata.getLastTransaction().getPosition());
    saveState(newState);
    stateHistory.add(newState);
    stateRollbackCount.set(1);
}
Also used : SourceState(com.airbnb.spinaltap.common.source.SourceState) MysqlMutation(com.airbnb.spinaltap.mysql.mutation.MysqlMutation) MysqlMutationMetadata(com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata)

Example 2 with MysqlMutationMetadata

use of com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata in project SpinalTap by airbnb.

the class MysqlDestinationMetrics method getTags.

@Override
protected Map<String, String> getTags(Mutation.Metadata meta) {
    Preconditions.checkState(meta instanceof MysqlMutationMetadata);
    MysqlMutationMetadata metadata = (MysqlMutationMetadata) meta;
    Map<String, String> metadataTags = new HashMap<>();
    metadataTags.put("database_name", metadata.getTable().getDatabase());
    metadataTags.put("table_name", metadata.getTable().getName());
    metadataTags.putAll(super.getTags(meta));
    return metadataTags;
}
Also used : MysqlMutationMetadata(com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata) HashMap(java.util.HashMap)

Example 3 with MysqlMutationMetadata

use of com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata in project SpinalTap by airbnb.

the class UpdateMutationMapper method mapEvent.

@Override
protected List<MysqlMutation> mapEvent(Table table, UpdateEvent event) {
    List<MysqlMutation> mutations = Lists.newArrayList();
    Collection<ColumnMetadata> cols = table.getColumns().values();
    List<Map.Entry<Serializable[], Serializable[]>> rows = event.getRows();
    for (int position = 0; position < rows.size(); position++) {
        MysqlMutationMetadata metadata = createMetadata(table, event, position);
        Row previousRow = new Row(table, zip(rows.get(position).getKey(), cols));
        Row newRow = new Row(table, zip(rows.get(position).getValue(), cols));
        // to retain invariant that a mutation captures changes to a single PK
        if (table.getPrimaryKey().isPresent() && !previousRow.getPrimaryKeyValue().equals(newRow.getPrimaryKeyValue())) {
            mutations.add(new MysqlDeleteMutation(metadata, previousRow));
            mutations.add(new MysqlInsertMutation(metadata, newRow));
        } else {
            mutations.add(new MysqlUpdateMutation(metadata, previousRow, newRow));
        }
    }
    ;
    return mutations;
}
Also used : MysqlDeleteMutation(com.airbnb.spinaltap.mysql.mutation.MysqlDeleteMutation) ColumnMetadata(com.airbnb.spinaltap.mysql.mutation.schema.ColumnMetadata) Serializable(java.io.Serializable) MysqlMutationMetadata(com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata) MysqlUpdateMutation(com.airbnb.spinaltap.mysql.mutation.MysqlUpdateMutation) MysqlMutation(com.airbnb.spinaltap.mysql.mutation.MysqlMutation) MysqlInsertMutation(com.airbnb.spinaltap.mysql.mutation.MysqlInsertMutation) Row(com.airbnb.spinaltap.mysql.mutation.schema.Row)

Example 4 with MysqlMutationMetadata

use of com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata in project SpinalTap by airbnb.

the class UpdateMutationMapper method map.

public Mutation map(MysqlUpdateMutation mutation) {
    MysqlMutationMetadata metadata = mutation.getMetadata();
    Mutation thriftMutation = new Mutation(MutationType.UPDATE, metadata.getTimestamp(), sourceId, metadata.getDataSource().getThriftDataSource(), createBinlogHeader(metadata, mutation.getType().getCode()), metadata.getTable().getThriftTable(), transformToEntity(mutation.getRow()));
    thriftMutation.setPreviousEntity(transformToEntity(mutation.getPreviousRow()));
    return thriftMutation;
}
Also used : MysqlMutationMetadata(com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata) Mutation(com.airbnb.jitney.event.spinaltap.v1.Mutation) MysqlUpdateMutation(com.airbnb.spinaltap.mysql.mutation.MysqlUpdateMutation)

Example 5 with MysqlMutationMetadata

use of com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata in project SpinalTap by airbnb.

the class KafkaDestinationTest method createMutation.

private Mutation createMutation(MutationType type) {
    Mapper<com.airbnb.spinaltap.Mutation<?>, ? extends TBase<?, ?>> thriftMutationMapper = ThriftMutationMapper.create("spinaltap");
    Table table = new Table(0L, TABLE, DATABASE, ImmutableList.of(new ColumnMetadata("id", ColumnDataType.LONGLONG, true, 0)), ImmutableList.of("id"));
    MysqlMutationMetadata metadata = new MysqlMutationMetadata(new DataSource(HOSTNAME, 0, "service"), new BinlogFilePos(), table, 0L, 0L, 0L, null, null, 0L, 0);
    Row row = new Row(table, ImmutableMap.of("id", new Column(new ColumnMetadata("id", ColumnDataType.LONGLONG, true, 0), 1L)));
    MysqlMutation mutation;
    switch(type) {
        case INSERT:
            mutation = new MysqlInsertMutation(metadata, row);
            break;
        case UPDATE:
            mutation = new MysqlUpdateMutation(metadata, row, row);
            break;
        case DELETE:
            mutation = new MysqlDeleteMutation(metadata, row);
            break;
        default:
            mutation = null;
    }
    return (Mutation) (thriftMutationMapper.map(mutation));
}
Also used : MysqlDeleteMutation(com.airbnb.spinaltap.mysql.mutation.MysqlDeleteMutation) ColumnMetadata(com.airbnb.spinaltap.mysql.mutation.schema.ColumnMetadata) MysqlMutationMetadata(com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata) Table(com.airbnb.spinaltap.mysql.mutation.schema.Table) MysqlUpdateMutation(com.airbnb.spinaltap.mysql.mutation.MysqlUpdateMutation) DataSource(com.airbnb.spinaltap.mysql.DataSource) MysqlMutation(com.airbnb.spinaltap.mysql.mutation.MysqlMutation) Column(com.airbnb.spinaltap.mysql.mutation.schema.Column) MysqlInsertMutation(com.airbnb.spinaltap.mysql.mutation.MysqlInsertMutation) BinlogFilePos(com.airbnb.spinaltap.mysql.BinlogFilePos) MysqlUpdateMutation(com.airbnb.spinaltap.mysql.mutation.MysqlUpdateMutation) Mutation(com.airbnb.jitney.event.spinaltap.v1.Mutation) MysqlMutation(com.airbnb.spinaltap.mysql.mutation.MysqlMutation) MysqlDeleteMutation(com.airbnb.spinaltap.mysql.mutation.MysqlDeleteMutation) MysqlInsertMutation(com.airbnb.spinaltap.mysql.mutation.MysqlInsertMutation) Row(com.airbnb.spinaltap.mysql.mutation.schema.Row)

Aggregations

MysqlMutationMetadata (com.airbnb.spinaltap.mysql.mutation.MysqlMutationMetadata)8 MysqlInsertMutation (com.airbnb.spinaltap.mysql.mutation.MysqlInsertMutation)4 MysqlMutation (com.airbnb.spinaltap.mysql.mutation.MysqlMutation)4 SourceState (com.airbnb.spinaltap.common.source.SourceState)3 MysqlUpdateMutation (com.airbnb.spinaltap.mysql.mutation.MysqlUpdateMutation)3 Row (com.airbnb.spinaltap.mysql.mutation.schema.Row)3 Mutation (com.airbnb.jitney.event.spinaltap.v1.Mutation)2 MysqlDeleteMutation (com.airbnb.spinaltap.mysql.mutation.MysqlDeleteMutation)2 ColumnMetadata (com.airbnb.spinaltap.mysql.mutation.schema.ColumnMetadata)2 Test (org.junit.Test)2 BinlogFilePos (com.airbnb.spinaltap.mysql.BinlogFilePos)1 DataSource (com.airbnb.spinaltap.mysql.DataSource)1 Column (com.airbnb.spinaltap.mysql.mutation.schema.Column)1 Table (com.airbnb.spinaltap.mysql.mutation.schema.Table)1 Serializable (java.io.Serializable)1 HashMap (java.util.HashMap)1