Search in sources :

Example 16 with Mutation

use of com.google.datastore.v1.Mutation in project simple-bigtable by spotify.

the class BigtableMutationImplTest method testWriteColumnAndTimestamp.

@Test
public void testWriteColumnAndTimestamp() throws Exception {
    final BigtableMutationImpl bigtableMutationImpl = (BigtableMutationImpl) this.bigtableMutation.write("family:qualifier", ByteString.copyFromUtf8("value"), 100L).write("family", "qualifier", ByteString.copyFromUtf8("value"), 100L);
    verifyGetMutateRowRequest();
    bigtableMutationImpl.execute();
    bigtableMutationImpl.executeAsync();
    assertEquals(2, bigtableMutationImpl.getMutateRowRequest().getMutationsCount());
    for (Mutation mutation : bigtableMutationImpl.getMutateRowRequest().getMutationsList()) {
        assertEquals(Mutation.MutationCase.SET_CELL, mutation.getMutationCase());
        assertEquals("family", mutation.getSetCell().getFamilyName());
        assertEquals("qualifier", mutation.getSetCell().getColumnQualifier().toStringUtf8());
        assertEquals("value", mutation.getSetCell().getValue().toStringUtf8());
        assertEquals(100L, mutation.getSetCell().getTimestampMicros());
        assertEquals(Mutation.DeleteFromRow.getDefaultInstance(), mutation.getDeleteFromRow());
        assertEquals(Mutation.DeleteFromFamily.getDefaultInstance(), mutation.getDeleteFromFamily());
        assertEquals(Mutation.DeleteFromColumn.getDefaultInstance(), mutation.getDeleteFromColumn());
    }
    verify(bigtableMock.getMockedDataClient()).mutateRow(bigtableMutation.getMutateRowRequest().build());
    verify(bigtableMock.getMockedDataClient()).mutateRowAsync(bigtableMutation.getMutateRowRequest().build());
    verifyNoMoreInteractions(bigtableMock.getMockedDataClient());
}
Also used : Mutation(com.google.bigtable.v2.Mutation) Test(org.junit.Test)

Example 17 with Mutation

use of com.google.datastore.v1.Mutation in project simple-bigtable by spotify.

the class BigtableMutationImplTest method testDeleteColumnFamily.

@Test
public void testDeleteColumnFamily() throws Exception {
    final BigtableMutationImpl bigtableMutationImpl = (BigtableMutationImpl) this.bigtableMutation.deleteColumnFamily("family");
    verifyGetMutateRowRequest();
    bigtableMutationImpl.execute();
    bigtableMutationImpl.executeAsync();
    assertEquals(1, bigtableMutationImpl.getMutateRowRequest().getMutationsCount());
    final Mutation mutation = bigtableMutationImpl.getMutateRowRequest().getMutations(0);
    assertEquals(Mutation.MutationCase.DELETE_FROM_FAMILY, mutation.getMutationCase());
    assertEquals("family", mutation.getDeleteFromFamily().getFamilyName());
    assertEquals(Mutation.DeleteFromRow.getDefaultInstance(), mutation.getDeleteFromRow());
    assertEquals(Mutation.DeleteFromColumn.getDefaultInstance(), mutation.getDeleteFromColumn());
    assertEquals(Mutation.SetCell.getDefaultInstance(), mutation.getSetCell());
    verify(bigtableMock.getMockedDataClient()).mutateRow(bigtableMutation.getMutateRowRequest().build());
    verify(bigtableMock.getMockedDataClient()).mutateRowAsync(bigtableMutation.getMutateRowRequest().build());
    verifyNoMoreInteractions(bigtableMock.getMockedDataClient());
}
Also used : Mutation(com.google.bigtable.v2.Mutation) Test(org.junit.Test)

Example 18 with Mutation

use of com.google.datastore.v1.Mutation in project simple-bigtable by spotify.

the class BigtableMutationImplTest method testDeleteCellsFromColumn.

@Test
public void testDeleteCellsFromColumn() throws Exception {
    final BigtableMutationImpl bigtableMutationImpl = (BigtableMutationImpl) this.bigtableMutation.deleteCellsFromColumn("family", "qualifier", Optional.empty(), Optional.empty()).deleteCellsFromColumn("family", "qualifier", Optional.of(100L), Optional.of(999L));
    verifyGetMutateRowRequest();
    bigtableMutationImpl.execute();
    bigtableMutationImpl.executeAsync();
    assertEquals(2, bigtableMutationImpl.getMutateRowRequest().getMutationsCount());
    Mutation mutation = bigtableMutationImpl.getMutateRowRequest().getMutations(0);
    assertEquals(Mutation.MutationCase.DELETE_FROM_COLUMN, mutation.getMutationCase());
    assertEquals("family", mutation.getDeleteFromColumn().getFamilyName());
    assertEquals("qualifier", mutation.getDeleteFromColumn().getColumnQualifier().toStringUtf8());
    assertEquals(TimestampRange.getDefaultInstance(), mutation.getDeleteFromColumn().getTimeRange());
    assertEquals(Mutation.DeleteFromRow.getDefaultInstance(), mutation.getDeleteFromRow());
    assertEquals(Mutation.DeleteFromFamily.getDefaultInstance(), mutation.getDeleteFromFamily());
    assertEquals(Mutation.SetCell.getDefaultInstance(), mutation.getSetCell());
    mutation = bigtableMutationImpl.getMutateRowRequest().getMutations(1);
    assertEquals(Mutation.MutationCase.DELETE_FROM_COLUMN, mutation.getMutationCase());
    assertEquals("family", mutation.getDeleteFromColumn().getFamilyName());
    assertEquals("qualifier", mutation.getDeleteFromColumn().getColumnQualifier().toStringUtf8());
    assertEquals(100L, mutation.getDeleteFromColumn().getTimeRange().getStartTimestampMicros());
    assertEquals(999L, mutation.getDeleteFromColumn().getTimeRange().getEndTimestampMicros());
    assertEquals(Mutation.DeleteFromRow.getDefaultInstance(), mutation.getDeleteFromRow());
    assertEquals(Mutation.DeleteFromFamily.getDefaultInstance(), mutation.getDeleteFromFamily());
    assertEquals(Mutation.SetCell.getDefaultInstance(), mutation.getSetCell());
    verify(bigtableMock.getMockedDataClient()).mutateRow(bigtableMutation.getMutateRowRequest().build());
    verify(bigtableMock.getMockedDataClient()).mutateRowAsync(bigtableMutation.getMutateRowRequest().build());
    verifyNoMoreInteractions(bigtableMock.getMockedDataClient());
}
Also used : Mutation(com.google.bigtable.v2.Mutation) Test(org.junit.Test)

Example 19 with Mutation

use of com.google.datastore.v1.Mutation in project beam by apache.

the class DatastoreV1Test method testDeleteEntities.

/**
 * Test that entities with valid keys are transformed to delete mutations.
 */
@Test
public void testDeleteEntities() throws Exception {
    Key key = makeKey("bird", "finch").build();
    Entity entity = Entity.newBuilder().setKey(key).build();
    DeleteEntityFn deleteEntityFn = new DeleteEntityFn();
    Mutation expectedMutation = makeDelete(entity.getKey()).build();
    assertEquals(expectedMutation, deleteEntityFn.apply(entity));
}
Also used : Entity(com.google.datastore.v1.Entity) DeleteEntity(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteEntity) DeleteEntityFn(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteEntityFn) Mutation(com.google.datastore.v1.Mutation) DeleteKey(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteKey) Key(com.google.datastore.v1.Key) DatastoreHelper.makeKey(com.google.datastore.v1.client.DatastoreHelper.makeKey) DatastoreV1.isValidKey(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.isValidKey) Test(org.junit.Test)

Example 20 with Mutation

use of com.google.datastore.v1.Mutation in project beam by apache.

the class DatastoreV1Test method testDatatoreWriterFnWithLargeEntities.

/**
 * Tests {@link DatastoreWriterFn} with large entities that need to be split into more batches.
 */
@Test
public void testDatatoreWriterFnWithLargeEntities() throws Exception {
    List<Mutation> mutations = new ArrayList<>();
    int entitySize = 0;
    for (int i = 0; i < 12; ++i) {
        Entity entity = Entity.newBuilder().setKey(makeKey("key" + i, i + 1)).putProperties("long", makeValue(new String(new char[900_000])).setExcludeFromIndexes(true).build()).build();
        // Take the size of any one entity.
        entitySize = entity.getSerializedSize();
        mutations.add(makeUpsert(entity).build());
    }
    DatastoreWriterFn datastoreWriter = new DatastoreWriterFn(StaticValueProvider.of(PROJECT_ID), null, mockDatastoreFactory, new FakeWriteBatcher());
    DoFnTester<Mutation, Void> doFnTester = DoFnTester.of(datastoreWriter);
    doFnTester.setCloningBehavior(CloningBehavior.DO_NOT_CLONE);
    doFnTester.processBundle(mutations);
    // This test is over-specific currently; it requires that we split the 12 entity writes into 3
    // requests, but we only need each CommitRequest to be less than 10MB in size.
    int entitiesPerRpc = DATASTORE_BATCH_UPDATE_BYTES_LIMIT / entitySize;
    int start = 0;
    while (start < mutations.size()) {
        int end = Math.min(mutations.size(), start + entitiesPerRpc);
        CommitRequest.Builder commitRequest = CommitRequest.newBuilder();
        commitRequest.setMode(CommitRequest.Mode.NON_TRANSACTIONAL);
        commitRequest.addAllMutations(mutations.subList(start, end));
        // Verify all the batch requests were made with the expected mutations.
        verify(mockDatastore).commit(commitRequest.build());
        start = end;
    }
}
Also used : CommitRequest(com.google.datastore.v1.CommitRequest) Entity(com.google.datastore.v1.Entity) DeleteEntity(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteEntity) ArrayList(java.util.ArrayList) Mutation(com.google.datastore.v1.Mutation) DatastoreWriterFn(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DatastoreWriterFn) Test(org.junit.Test)

Aggregations

Test (org.junit.Test)19 Mutation (com.google.bigtable.v2.Mutation)13 Mutation (com.google.datastore.v1.Mutation)7 Mutation (com.airbnb.jitney.event.spinaltap.v1.Mutation)5 ByteString (com.google.protobuf.ByteString)5 ArrayList (java.util.ArrayList)5 MysqlUpdateMutation (com.airbnb.spinaltap.mysql.mutation.MysqlUpdateMutation)4 Key (com.google.datastore.v1.Key)4 DatastoreHelper.makeKey (com.google.datastore.v1.client.DatastoreHelper.makeKey)4 MysqlDeleteMutation (com.airbnb.spinaltap.mysql.mutation.MysqlDeleteMutation)3 MysqlInsertMutation (com.airbnb.spinaltap.mysql.mutation.MysqlInsertMutation)3 MysqlMutation (com.airbnb.spinaltap.mysql.mutation.MysqlMutation)3 CommitRequest (com.google.datastore.v1.CommitRequest)3 Entity (com.google.datastore.v1.Entity)3 DatastoreException (com.google.datastore.v1.client.DatastoreException)3 DatastoreWriterFn (org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DatastoreWriterFn)3 DeleteEntity (org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteEntity)3 DeleteKey (org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteKey)3 DatastoreV1.isValidKey (org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.isValidKey)3 KV (org.apache.beam.sdk.values.KV)3