Search in sources :

Example 1 with Builder

use of com.google.firestore.v1.Write.Builder in project spring-cloud-gcp by spring-cloud.

the class PartTreeFirestoreQuery method createBuilderWithFilter.

private StructuredQuery.Builder createBuilderWithFilter(Object[] parameters) {
    StructuredQuery.Builder builder = StructuredQuery.newBuilder();
    Iterator it = Arrays.asList(parameters).iterator();
    StructuredQuery.CompositeFilter.Builder compositeFilter = StructuredQuery.CompositeFilter.newBuilder();
    compositeFilter.setOp(StructuredQuery.CompositeFilter.Operator.AND);
    this.tree.getParts().forEach(part -> {
        StructuredQuery.FieldReference fieldReference = StructuredQuery.FieldReference.newBuilder().setFieldPath(buildName(part)).build();
        StructuredQuery.Filter.Builder filter = StructuredQuery.Filter.newBuilder();
        if (part.getType() == Part.Type.IS_NULL) {
            filter.getUnaryFilterBuilder().setField(fieldReference).setOp(StructuredQuery.UnaryFilter.Operator.IS_NULL);
        } else {
            if (!it.hasNext()) {
                throw new FirestoreDataException("Too few parameters are provided for query method: " + getQueryMethod().getName());
            }
            Object value = it.next();
            filter.getFieldFilterBuilder().setField(fieldReference).setOp(getOperator(part, value)).setValue(this.classMapper.toFirestoreValue(value));
        }
        compositeFilter.addFilters(filter.build());
    });
    builder.setWhere(StructuredQuery.Filter.newBuilder().setCompositeFilter(compositeFilter.build()));
    return builder;
}
Also used : StructuredQuery(com.google.firestore.v1.StructuredQuery) Iterator(java.util.Iterator) FirestoreDataException(org.springframework.cloud.gcp.data.firestore.FirestoreDataException) FieldReference(com.google.firestore.v1.StructuredQuery.FieldReference)

Example 2 with Builder

use of com.google.firestore.v1.Write.Builder in project spring-cloud-gcp by spring-cloud.

the class PartTreeFirestoreQuery method execute.

@Override
public Object execute(Object[] parameters) {
    StructuredQuery.Builder builder = createBuilderWithFilter(parameters);
    // Handle Pageable parameters.
    if (!getQueryMethod().getParameters().isEmpty()) {
        ParameterAccessor paramAccessor = new ParametersParameterAccessor(getQueryMethod().getParameters(), parameters);
        Pageable pageable = paramAccessor.getPageable();
        if (pageable != null && pageable.isPaged()) {
            builder.setOffset((int) Math.min(Integer.MAX_VALUE, pageable.getOffset()));
            builder.setLimit(Int32Value.newBuilder().setValue(pageable.getPageSize()));
        }
        Sort sort = paramAccessor.getSort();
        if (sort != null) {
            builder.addAllOrderBy(createFirestoreSortOrders(sort));
        }
    }
    if (this.tree.isCountProjection()) {
        return this.reactiveOperations.count(this.persistentEntity.getType(), builder);
    } else {
        return this.reactiveOperations.execute(builder, this.persistentEntity.getType());
    }
}
Also used : StructuredQuery(com.google.firestore.v1.StructuredQuery) Pageable(org.springframework.data.domain.Pageable) ParametersParameterAccessor(org.springframework.data.repository.query.ParametersParameterAccessor) ParametersParameterAccessor(org.springframework.data.repository.query.ParametersParameterAccessor) ParameterAccessor(org.springframework.data.repository.query.ParameterAccessor) Sort(org.springframework.data.domain.Sort)

Example 3 with Builder

use of com.google.firestore.v1.Write.Builder in project spring-cloud-gcp by spring-cloud.

the class FirestoreTemplate method createUpdateWrite.

private <T> Write createUpdateWrite(T entity) {
    Builder builder = Write.newBuilder();
    if (getIdValue(entity) == null) {
        builder.setCurrentDocument(Precondition.newBuilder().setExists(false).build());
    }
    String resourceName = buildResourceName(entity);
    Document document = getClassMapper().entityToDocument(entity, resourceName);
    return builder.setUpdate(document).build();
}
Also used : Builder(com.google.firestore.v1.Write.Builder) Document(com.google.firestore.v1.Document)

Example 4 with Builder

use of com.google.firestore.v1.Write.Builder in project beam by apache.

the class BaseFirestoreV1WriteFnTest method endToEnd_deadlineExceededOnAnIndividualWriteResultsInThrottling.

@Test
public final void endToEnd_deadlineExceededOnAnIndividualWriteResultsInThrottling() throws Exception {
    final long totalDocCount = 1_000_000;
    final int numWorkers = 100;
    final long docCount = totalDocCount / numWorkers;
    LOG.info("docCount = {}", docCount);
    RpcQosOptions options = rpcQosOptions.toBuilder().withHintMaxNumWorkers(numWorkers).withSamplePeriod(Duration.standardMinutes(10)).withReportDiagnosticMetrics().build();
    LOG.debug("options = {}", options);
    FirestoreStatefulComponentFactory ff = mock(FirestoreStatefulComponentFactory.class);
    when(ff.getFirestoreStub(any())).thenReturn(stub);
    Random random = new Random(12345);
    TestClock clock = new TestClock(Instant.EPOCH, Duration.standardSeconds(1));
    Sleeper sleeper = millis -> clock.setNext(advanceClockBy(Duration.millis(millis)));
    RpcQosImpl qos = new RpcQosImpl(options, random, sleeper, metricsFixture.counterFactory, metricsFixture.distributionFactory);
    RpcQos qosSpy = mock(RpcQos.class, invocation -> {
        Method method = invocation.getMethod();
        LOG.debug("method = {}", method);
        Method actualMethod = qos.getClass().getMethod(method.getName(), method.getParameterTypes());
        return actualMethod.invoke(qos, invocation.getArguments());
    });
    when(ff.getRpcQos(options)).thenReturn(qosSpy);
    int defaultDocumentWriteLatency = 30;
    final AtomicLong writeCounter = new AtomicLong();
    when(processContext.element()).thenAnswer(invocation -> newWrite(writeCounter.getAndIncrement()));
    when(callable.call(any())).thenAnswer(new Answer<BatchWriteResponse>() {

        private final Random rand = new Random(84572908);

        private final Instant threshold = Instant.ofEpochMilli(Duration.standardMinutes(20).getMillis());

        @Override
        public BatchWriteResponse answer(InvocationOnMock invocation) throws Throwable {
            BatchWriteRequest request = invocation.getArgument(0, BatchWriteRequest.class);
            LOG.debug("request = {}", request);
            long requestDurationMs = 0;
            BatchWriteResponse.Builder builder = BatchWriteResponse.newBuilder();
            for (Write ignored : request.getWritesList()) {
                builder.addWriteResults(WriteResult.newBuilder().build());
                if (clock.prev.isBefore(threshold)) {
                    requestDurationMs += defaultDocumentWriteLatency;
                    builder.addStatus(STATUS_OK);
                } else {
                    int latency = rand.nextInt(1500);
                    LOG.debug("latency = {}", latency);
                    if (latency > 300) {
                        builder.addStatus(STATUS_DEADLINE_EXCEEDED);
                    } else {
                        builder.addStatus(STATUS_OK);
                    }
                    requestDurationMs += latency;
                }
            }
            clock.setNext(advanceClockBy(Duration.millis(requestDurationMs)));
            return builder.build();
        }
    });
    LOG.info("### parameters: {defaultDocumentWriteLatency: {}, rpcQosOptions: {}}", defaultDocumentWriteLatency, options);
    FnT fn = getFn(clock, ff, options, metricsFixture.counterFactory, metricsFixture.distributionFactory);
    fn.setup();
    fn.startBundle(startBundleContext);
    while (writeCounter.get() < docCount) {
        fn.processElement(processContext, window);
    }
    fn.finishBundle(finishBundleContext);
    LOG.info("writeCounter = {}", writeCounter.get());
    LOG.info("clock.prev = {}", clock.prev);
    MyDistribution qosAdaptiveThrottlerThrottlingMs = metricsFixture.distributions.get("qos_adaptiveThrottler_throttlingMs");
    assertNotNull(qosAdaptiveThrottlerThrottlingMs);
    List<Long> updateInvocations = qosAdaptiveThrottlerThrottlingMs.updateInvocations;
    assertFalse(updateInvocations.isEmpty());
}
Also used : MetricName(org.apache.beam.sdk.metrics.MetricName) BatchWriteRequest(com.google.firestore.v1.BatchWriteRequest) Write(com.google.firestore.v1.Write) ArgumentMatchers.eq(org.mockito.ArgumentMatchers.eq) LoggerFactory(org.slf4j.LoggerFactory) Random(java.util.Random) ImmutableMap(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap) Element(org.apache.beam.sdk.io.gcp.firestore.RpcQos.RpcWriteAttempt.Element) Mockito.doThrow(org.mockito.Mockito.doThrow) Lists.newArrayList(org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList) Mockito.verifyNoMoreInteractions(org.mockito.Mockito.verifyNoMoreInteractions) Map(java.util.Map) Assert.fail(org.junit.Assert.fail) Method(java.lang.reflect.Method) WriteElement(org.apache.beam.sdk.io.gcp.firestore.FirestoreV1WriteFn.WriteElement) WriteResult(com.google.firestore.v1.WriteResult) Mockito.atLeastOnce(org.mockito.Mockito.atLeastOnce) Status(com.google.rpc.Status) Sleeper(org.apache.beam.sdk.util.Sleeper) Mockito.doNothing(org.mockito.Mockito.doNothing) List(java.util.List) Assert.assertFalse(org.junit.Assert.assertFalse) Mockito.mock(org.mockito.Mockito.mock) ArgumentMatchers.any(org.mockito.ArgumentMatchers.any) HasRpcAttemptContext(org.apache.beam.sdk.io.gcp.firestore.FirestoreV1RpcAttemptContexts.HasRpcAttemptContext) RpcWriteAttempt(org.apache.beam.sdk.io.gcp.firestore.RpcQos.RpcWriteAttempt) Mock(org.mockito.Mock) ApiExceptionFactory(com.google.api.gax.rpc.ApiExceptionFactory) Duration(org.joda.time.Duration) HashMap(java.util.HashMap) Function(java.util.function.Function) ArrayList(java.util.ArrayList) Assert.assertSame(org.junit.Assert.assertSame) Answer(org.mockito.stubbing.Answer) InvocationOnMock(org.mockito.invocation.InvocationOnMock) ArgumentCaptor(org.mockito.ArgumentCaptor) FlushBuffer(org.apache.beam.sdk.io.gcp.firestore.RpcQos.RpcWriteAttempt.FlushBuffer) GrpcStatusCode(com.google.api.gax.grpc.GrpcStatusCode) BatchWriteResponse(com.google.firestore.v1.BatchWriteResponse) Code(com.google.rpc.Code) ArgumentMatchers.anyInt(org.mockito.ArgumentMatchers.anyInt) Before(org.junit.Before) DoFn(org.apache.beam.sdk.transforms.DoFn) Logger(org.slf4j.Logger) AdditionalMatchers.gt(org.mockito.AdditionalMatchers.gt) Assert.assertNotNull(org.junit.Assert.assertNotNull) Counter(org.apache.beam.sdk.metrics.Counter) Assert.assertTrue(org.junit.Assert.assertTrue) BaseBatchWriteFn(org.apache.beam.sdk.io.gcp.firestore.FirestoreV1WriteFn.BaseBatchWriteFn) Mockito.times(org.mockito.Mockito.times) IOException(java.io.IOException) FlushBufferImpl(org.apache.beam.sdk.io.gcp.firestore.RpcQosImpl.FlushBufferImpl) Test(org.junit.Test) Mockito.when(org.mockito.Mockito.when) ApiException(com.google.api.gax.rpc.ApiException) Mockito.verify(org.mockito.Mockito.verify) Distribution(org.apache.beam.sdk.metrics.Distribution) UnaryCallable(com.google.api.gax.rpc.UnaryCallable) AtomicLong(java.util.concurrent.atomic.AtomicLong) Mockito.never(org.mockito.Mockito.never) PowerMockito.spy(org.powermock.api.mockito.PowerMockito.spy) FirestoreProtoHelpers.newWrite(org.apache.beam.sdk.io.gcp.firestore.FirestoreProtoHelpers.newWrite) BoundedWindow(org.apache.beam.sdk.transforms.windowing.BoundedWindow) Instant(org.joda.time.Instant) Assert.assertEquals(org.junit.Assert.assertEquals) Write(com.google.firestore.v1.Write) FirestoreProtoHelpers.newWrite(org.apache.beam.sdk.io.gcp.firestore.FirestoreProtoHelpers.newWrite) BatchWriteResponse(com.google.firestore.v1.BatchWriteResponse) Sleeper(org.apache.beam.sdk.util.Sleeper) Random(java.util.Random) BatchWriteRequest(com.google.firestore.v1.BatchWriteRequest) Instant(org.joda.time.Instant) Method(java.lang.reflect.Method) AtomicLong(java.util.concurrent.atomic.AtomicLong) InvocationOnMock(org.mockito.invocation.InvocationOnMock) AtomicLong(java.util.concurrent.atomic.AtomicLong) Test(org.junit.Test)

Example 5 with Builder

use of com.google.firestore.v1.Write.Builder in project beam by apache.

the class BaseFirestoreV1WriteFnTest method endToEnd_maxBatchSizeRespected.

@Test
public final void endToEnd_maxBatchSizeRespected() throws Exception {
    Instant enqueue0 = Instant.ofEpochMilli(0);
    Instant enqueue1 = Instant.ofEpochMilli(1);
    Instant enqueue2 = Instant.ofEpochMilli(2);
    Instant enqueue3 = Instant.ofEpochMilli(3);
    Instant enqueue4 = Instant.ofEpochMilli(4);
    Instant group1Rpc1Start = Instant.ofEpochMilli(5);
    Instant group1Rpc1End = Instant.ofEpochMilli(6);
    Instant enqueue5 = Instant.ofEpochMilli(7);
    Instant finalFlush = Instant.ofEpochMilli(8);
    Instant group2Rpc1Start = Instant.ofEpochMilli(9);
    Instant group2Rpc1End = Instant.ofEpochMilli(10);
    Write write0 = newWrite(0);
    Write write1 = newWrite(1);
    Write write2 = newWrite(2);
    Write write3 = newWrite(3);
    Write write4 = newWrite(4);
    Write write5 = newWrite(5);
    int maxValuesPerGroup = 5;
    BatchWriteRequest.Builder builder = BatchWriteRequest.newBuilder().setDatabase("projects/testing-project/databases/(default)");
    BatchWriteRequest expectedGroup1Request = builder.build().toBuilder().addWrites(write0).addWrites(write1).addWrites(write2).addWrites(write3).addWrites(write4).build();
    BatchWriteRequest expectedGroup2Request = builder.build().toBuilder().addWrites(write5).build();
    BatchWriteResponse group1Response = BatchWriteResponse.newBuilder().addStatus(STATUS_OK).addStatus(STATUS_OK).addStatus(STATUS_OK).addStatus(STATUS_OK).addStatus(STATUS_OK).build();
    BatchWriteResponse group2Response = BatchWriteResponse.newBuilder().addStatus(STATUS_OK).build();
    RpcQosOptions options = rpcQosOptions.toBuilder().withBatchMaxCount(maxValuesPerGroup).build();
    FlushBuffer<Element<Write>> flushBuffer = spy(newFlushBuffer(options));
    FlushBuffer<Element<Write>> flushBuffer2 = spy(newFlushBuffer(options));
    when(processContext.element()).thenReturn(write0, write1, write2, write3, write4, write5);
    when(rpcQos.newWriteAttempt(any())).thenReturn(attempt, attempt, attempt, attempt, attempt, attempt2, attempt2, attempt2).thenThrow(new IllegalStateException("too many attempts"));
    when(attempt.awaitSafeToProceed(any())).thenReturn(true);
    when(attempt2.awaitSafeToProceed(any())).thenReturn(true);
    when(attempt.<Write, Element<Write>>newFlushBuffer(enqueue0)).thenReturn(newFlushBuffer(options));
    when(attempt.<Write, Element<Write>>newFlushBuffer(enqueue1)).thenReturn(newFlushBuffer(options));
    when(attempt.<Write, Element<Write>>newFlushBuffer(enqueue2)).thenReturn(newFlushBuffer(options));
    when(attempt.<Write, Element<Write>>newFlushBuffer(enqueue3)).thenReturn(newFlushBuffer(options));
    when(attempt.<Write, Element<Write>>newFlushBuffer(enqueue4)).thenReturn(flushBuffer);
    when(callable.call(expectedGroup1Request)).thenReturn(group1Response);
    when(attempt2.<Write, Element<Write>>newFlushBuffer(enqueue5)).thenReturn(newFlushBuffer(options));
    when(attempt2.<Write, Element<Write>>newFlushBuffer(finalFlush)).thenReturn(flushBuffer2);
    when(callable.call(expectedGroup2Request)).thenReturn(group2Response);
    runFunction(getFn(clock, ff, options, CounterFactory.DEFAULT, DistributionFactory.DEFAULT), maxValuesPerGroup + 1);
    verify(attempt, times(1)).recordRequestStart(group1Rpc1Start, 5);
    verify(attempt, times(1)).recordWriteCounts(group1Rpc1End, 5, 0);
    verify(attempt, times(1)).completeSuccess();
    verify(attempt2, times(1)).recordRequestStart(group2Rpc1Start, 1);
    verify(attempt2, times(1)).recordWriteCounts(group2Rpc1End, 1, 0);
    verify(attempt2, times(1)).completeSuccess();
    verify(callable, times(1)).call(expectedGroup1Request);
    verify(callable, times(1)).call(expectedGroup2Request);
    verifyNoMoreInteractions(callable);
    verify(flushBuffer, times(maxValuesPerGroup)).offer(any());
    verify(flushBuffer2, times(1)).offer(any());
}
Also used : Write(com.google.firestore.v1.Write) FirestoreProtoHelpers.newWrite(org.apache.beam.sdk.io.gcp.firestore.FirestoreProtoHelpers.newWrite) Instant(org.joda.time.Instant) Element(org.apache.beam.sdk.io.gcp.firestore.RpcQos.RpcWriteAttempt.Element) WriteElement(org.apache.beam.sdk.io.gcp.firestore.FirestoreV1WriteFn.WriteElement) BatchWriteRequest(com.google.firestore.v1.BatchWriteRequest) BatchWriteResponse(com.google.firestore.v1.BatchWriteResponse) Test(org.junit.Test)

Aggregations

StructuredQuery (com.google.firestore.v1.StructuredQuery)3 BatchWriteRequest (com.google.firestore.v1.BatchWriteRequest)2 BatchWriteResponse (com.google.firestore.v1.BatchWriteResponse)2 Write (com.google.firestore.v1.Write)2 FirestoreProtoHelpers.newWrite (org.apache.beam.sdk.io.gcp.firestore.FirestoreProtoHelpers.newWrite)2 WriteElement (org.apache.beam.sdk.io.gcp.firestore.FirestoreV1WriteFn.WriteElement)2 Element (org.apache.beam.sdk.io.gcp.firestore.RpcQos.RpcWriteAttempt.Element)2 Test (org.junit.Test)2 GrpcStatusCode (com.google.api.gax.grpc.GrpcStatusCode)1 ApiException (com.google.api.gax.rpc.ApiException)1 ApiExceptionFactory (com.google.api.gax.rpc.ApiExceptionFactory)1 UnaryCallable (com.google.api.gax.rpc.UnaryCallable)1 Document (com.google.firestore.v1.Document)1 RunQueryRequest (com.google.firestore.v1.RunQueryRequest)1 FieldReference (com.google.firestore.v1.StructuredQuery.FieldReference)1 Builder (com.google.firestore.v1.Write.Builder)1 WriteResult (com.google.firestore.v1.WriteResult)1 Code (com.google.rpc.Code)1 Status (com.google.rpc.Status)1 IOException (java.io.IOException)1