Search in sources :

Example 1 with Query

use of org.hypertrace.entity.data.service.v1.Query in project beam by apache.

the class DataStoreReadWriteIT method testWriteRead_viaCoreBeamIO.

@Test
public void testWriteRead_viaCoreBeamIO() {
    String projectId = options.getProject();
    Key ancestor = makeKey(KIND, UUID.randomUUID().toString()).build();
    Key itemKey = makeKey(ancestor, KIND, UUID.randomUUID().toString()).setPartitionId(PartitionId.newBuilder().setProjectId(projectId).build()).build();
    Row testWriteRow = Row.withSchema(SOURCE_SCHEMA).addValues(itemKey.toByteArray(), "4000").build();
    writePipeline.apply(Create.of(testWriteRow).withRowSchema(SOURCE_SCHEMA)).apply(RowToEntity.create("__key__", KIND)).apply(DatastoreIO.v1().write().withProjectId(projectId));
    writePipeline.run().waitUntilFinish();
    Query.Builder query = Query.newBuilder();
    query.addKindBuilder().setName(KIND);
    query.setFilter(makeFilter("__key__", Operator.EQUAL, makeValue(itemKey)));
    DatastoreV1.Read read = DatastoreIO.v1().read().withProjectId(projectId).withQuery(query.build());
    PCollection<Row> rowsRead = readPipeline.apply(read).apply(EntityToRow.create(SOURCE_SCHEMA, "__key__"));
    PAssert.that(rowsRead).containsInAnyOrder(testWriteRow);
    readPipeline.run().waitUntilFinish();
}
Also used : Query(com.google.datastore.v1.Query) DatastoreV1(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1) ByteString(org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.avatica.util.ByteString) Row(org.apache.beam.sdk.values.Row) EntityToRow(org.apache.beam.sdk.io.gcp.datastore.EntityToRow) Key(com.google.datastore.v1.Key) DatastoreHelper.makeKey(com.google.datastore.v1.client.DatastoreHelper.makeKey) Test(org.junit.Test)

Example 2 with Query

use of org.hypertrace.entity.data.service.v1.Query in project beam by apache.

the class SplitQueryFnIT method testSplitQueryFn.

/**
 * A helper method to test {@link SplitQueryFn} to generate the expected number of splits.
 */
private void testSplitQueryFn(String projectId, String kind, @Nullable String namespace, int expectedNumSplits) throws Exception {
    Query.Builder query = Query.newBuilder();
    query.addKindBuilder().setName(kind);
    SplitQueryFn splitQueryFn = new SplitQueryFn(V1Options.from(projectId, namespace, null), 0);
    DoFnTester<Query, Query> doFnTester = DoFnTester.of(splitQueryFn);
    List<Query> queries = doFnTester.processBundle(query.build());
    assertEquals(expectedNumSplits, queries.size());
}
Also used : Query(com.google.datastore.v1.Query) SplitQueryFn(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.Read.SplitQueryFn)

Example 3 with Query

use of org.hypertrace.entity.data.service.v1.Query in project beam by apache.

the class DatastoreV1Test method testReadValidationFailsQueryLimitZero.

@Test
public void testReadValidationFailsQueryLimitZero() throws Exception {
    Query invalidLimit = Query.newBuilder().setLimit(Int32Value.newBuilder().setValue(0)).build();
    thrown.expect(IllegalArgumentException.class);
    thrown.expectMessage("Invalid query limit 0: must be positive");
    DatastoreIO.v1().read().withQuery(invalidLimit);
}
Also used : GqlQuery(com.google.datastore.v1.GqlQuery) Query(com.google.datastore.v1.Query) Test(org.junit.Test)

Example 4 with Query

use of org.hypertrace.entity.data.service.v1.Query in project beam by apache.

the class DatastoreV1Test method testSplitQueryFnWithQueryLimit.

/**
 * Tests {@link DatastoreV1.Read.SplitQueryFn} when the query has a user specified limit.
 */
@Test
public void testSplitQueryFnWithQueryLimit() throws Exception {
    Query queryWithLimit = QUERY.toBuilder().setLimit(Int32Value.newBuilder().setValue(1)).build();
    SplitQueryFn splitQueryFn = new SplitQueryFn(V_1_OPTIONS, 10, mockDatastoreFactory);
    DoFnTester<Query, Query> doFnTester = DoFnTester.of(splitQueryFn);
    doFnTester.setCloningBehavior(CloningBehavior.DO_NOT_CLONE);
    List<Query> queries = doFnTester.processBundle(queryWithLimit);
    assertEquals(1, queries.size());
    verifyNoMoreInteractions(mockDatastore);
    verifyNoMoreInteractions(mockQuerySplitter);
}
Also used : GqlQuery(com.google.datastore.v1.GqlQuery) Query(com.google.datastore.v1.Query) SplitQueryFn(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.Read.SplitQueryFn) Test(org.junit.Test)

Example 5 with Query

use of org.hypertrace.entity.data.service.v1.Query in project beam by apache.

the class DatastoreV1Test method testSplitQueryFnWithNumSplits.

/**
 * Tests {@link SplitQueryFn} when number of query splits is specified.
 */
@Test
public void testSplitQueryFnWithNumSplits() throws Exception {
    int numSplits = 100;
    when(mockQuerySplitter.getSplits(eq(QUERY), any(PartitionId.class), eq(numSplits), any(Datastore.class))).thenReturn(splitQuery(QUERY, numSplits));
    SplitQueryFn splitQueryFn = new SplitQueryFn(V_1_OPTIONS, numSplits, mockDatastoreFactory);
    DoFnTester<Query, Query> doFnTester = DoFnTester.of(splitQueryFn);
    /**
     * Although Datastore client is marked transient in {@link SplitQueryFn}, when injected through
     * mock factory using a when clause for unit testing purposes, it is not serializable because it
     * doesn't have a no-arg constructor. Thus disabling the cloning to prevent the doFn from being
     * serialized.
     */
    doFnTester.setCloningBehavior(CloningBehavior.DO_NOT_CLONE);
    List<Query> queries = doFnTester.processBundle(QUERY);
    assertEquals(queries.size(), numSplits);
    // Confirms that sub-queries are not equal to original when there is more than one split.
    for (Query subQuery : queries) {
        assertNotEquals(subQuery, QUERY);
    }
    verify(mockQuerySplitter, times(1)).getSplits(eq(QUERY), any(PartitionId.class), eq(numSplits), any(Datastore.class));
    verifyZeroInteractions(mockDatastore);
}
Also used : Datastore(com.google.datastore.v1.client.Datastore) GqlQuery(com.google.datastore.v1.GqlQuery) Query(com.google.datastore.v1.Query) SplitQueryFn(org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.Read.SplitQueryFn) PartitionId(com.google.datastore.v1.PartitionId) Test(org.junit.Test)

Aggregations

Query (org.jpl7.Query)88 Term (org.jpl7.Term)52 Variable (org.jpl7.Variable)32 Query (org.hypertrace.entity.data.service.v1.Query)31 Map (java.util.Map)27 Test (org.junit.jupiter.api.Test)27 Atom (org.jpl7.Atom)21 Compound (org.jpl7.Compound)18 Filter (org.hypertrace.core.documentstore.Filter)17 Test (org.junit.Test)16 AttributeFilter (org.hypertrace.entity.data.service.v1.AttributeFilter)15 Query (com.google.datastore.v1.Query)14 ArrayList (java.util.ArrayList)10 Entity (org.hypertrace.entity.data.service.v1.Entity)9 EntityQueryRequest (org.hypertrace.entity.query.service.v1.EntityQueryRequest)8 GqlQuery (com.google.datastore.v1.GqlQuery)7 Integer (org.jpl7.Integer)6 IOException (java.io.IOException)5 Collections (java.util.Collections)5 List (java.util.List)5