Search in sources :

Example 16 with Query

use of org.skife.jdbi.v2.Query in project beam by apache.

the class V1ReadIT method testE2EV1Read.

/**
   * An end-to-end test for {@link DatastoreV1.Read#withQuery(Query)}
   *
   * <p>Write some test entities to datastore and then run a pipeline that
   * reads and counts the total number of entities. Verify that the count matches the
   * number of entities written.
   */
@Test
public void testE2EV1Read() throws Exception {
    // Read from datastore
    Query query = V1TestUtil.makeAncestorKindQuery(options.getKind(), options.getNamespace(), ancestor);
    DatastoreV1.Read read = DatastoreIO.v1().read().withProjectId(project).withQuery(query).withNamespace(options.getNamespace());
    // Count the total number of entities
    Pipeline p = Pipeline.create(options);
    PCollection<Long> count = p.apply(read).apply(Count.<Entity>globally());
    PAssert.thatSingleton(count).isEqualTo(numEntities);
    p.run();
}
Also used : Query(com.google.datastore.v1.Query) TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Pipeline(org.apache.beam.sdk.Pipeline) Test(org.junit.Test)

Example 17 with Query

use of org.skife.jdbi.v2.Query in project beam by apache.

the class DatastoreV1Test method readFnTest.

/** Helper function to run a test reading from a {@link ReadFn}. */
private void readFnTest(int numEntities) throws Exception {
    // An empty query to read entities.
    Query query = Query.newBuilder().setLimit(Int32Value.newBuilder().setValue(numEntities)).build();
    // Use mockResponseForQuery to generate results.
    when(mockDatastore.runQuery(any(RunQueryRequest.class))).thenAnswer(new Answer<RunQueryResponse>() {

        @Override
        public RunQueryResponse answer(InvocationOnMock invocationOnMock) throws Throwable {
            Query q = ((RunQueryRequest) invocationOnMock.getArguments()[0]).getQuery();
            return mockResponseForQuery(q);
        }
    });
    ReadFn readFn = new ReadFn(V_1_OPTIONS, mockDatastoreFactory);
    DoFnTester<Query, Entity> doFnTester = DoFnTester.of(readFn);
    /**
     * Although Datastore client is marked transient in {@link ReadFn}, when injected through
     * mock factory using a when clause for unit testing purposes, it is not serializable
     * because it doesn't have a no-arg constructor. Thus disabling the cloning to prevent the
     * test object from being serialized.
     */
    doFnTester.setCloningBehavior(CloningBehavior.DO_NOT_CLONE);
    List<Entity> entities = doFnTester.processBundle(query);
    int expectedNumCallsToRunQuery = (int) Math.ceil((double) numEntities / QUERY_BATCH_LIMIT);
    verify(mockDatastore, times(expectedNumCallsToRunQuery)).runQuery(any(RunQueryRequest.class));
    // Validate the number of results.
    assertEquals(numEntities, entities.size());
}
Also used : Entity(com.google.datastore.v1.Entity) DeleteEntity(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.DeleteEntity) GqlQuery(com.google.datastore.v1.GqlQuery) Query(com.google.datastore.v1.Query) ReadFn(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.Read.ReadFn) RunQueryResponse(com.google.datastore.v1.RunQueryResponse) InvocationOnMock(org.mockito.invocation.InvocationOnMock) RunQueryRequest(com.google.datastore.v1.RunQueryRequest)

Aggregations

Query (com.google.datastore.v1.Query)10 Test (org.junit.Test)8 Handle (org.skife.jdbi.v2.Handle)7 GqlQuery (com.google.datastore.v1.GqlQuery)6 Datastore (com.google.datastore.v1.client.Datastore)4 SplitQueryFn (org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.Read.SplitQueryFn)4 KV (org.apache.beam.sdk.values.KV)4 SQLException (java.sql.SQLException)3 Entity (com.google.datastore.v1.Entity)2 PartitionId (com.google.datastore.v1.PartitionId)2 RunQueryRequest (com.google.datastore.v1.RunQueryRequest)2 RunQueryResponse (com.google.datastore.v1.RunQueryResponse)2 ResultSet (java.sql.ResultSet)2 ArrayList (java.util.ArrayList)2 List (java.util.List)2 Map (java.util.Map)2 DBI (org.skife.jdbi.v2.DBI)2 StatementContext (org.skife.jdbi.v2.StatementContext)2 Job (com.airbnb.airpal.api.Job)1 JobTableOutputJoinRow (com.airbnb.airpal.sql.beans.JobTableOutputJoinRow)1