Search in sources :

Example 11 with BatchStage

use of com.hazelcast.jet.pipeline.BatchStage in project hazelcast by hazelcast.

the class GrpcServiceTest method whenNotAsync_unary_distributed.

@Test
public void whenNotAsync_unary_distributed() throws IOException {
    // Given
    server = createServer(new GreeterServiceImpl());
    final int port = server.getPort();
    List<String> items = IntStream.range(0, ITEM_COUNT).mapToObj(Integer::toString).collect(toList());
    Pipeline p = Pipeline.create();
    BatchStageWithKey<String, String> stage = p.readFrom(TestSources.items(items)).groupingKey(i -> i);
    // When
    BatchStage<String> mapped = stage.mapUsingService(unary(port), (service, key, item) -> {
        HelloRequest req = HelloRequest.newBuilder().setName(item).build();
        return service.call(req).thenApply(HelloReply::getMessage).get();
    });
    // Then
    mapped.writeTo(AssertionSinks.assertCollected(e -> {
        assertEquals("unexpected number of items received", ITEM_COUNT, e.size());
    }));
    instance().getJet().newJob(p).join();
}
Also used : AssertionSinks(com.hazelcast.jet.pipeline.test.AssertionSinks) IntStream(java.util.stream.IntStream) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) BindableService(io.grpc.BindableService) Arrays(java.util.Arrays) BeforeClass(org.junit.BeforeClass) QuickTest(com.hazelcast.test.annotation.QuickTest) Assertions.assertThat(org.assertj.core.api.Assertions.assertThat) BatchStage(com.hazelcast.jet.pipeline.BatchStage) StreamObserver(io.grpc.stub.StreamObserver) HelloReply(com.hazelcast.jet.grpc.greeter.GreeterOuterClass.HelloReply) ServerBuilder(io.grpc.ServerBuilder) After(org.junit.After) GreeterGrpc(com.hazelcast.jet.grpc.greeter.GreeterGrpc) ServiceFactory(com.hazelcast.jet.pipeline.ServiceFactory) Assert.fail(org.junit.Assert.fail) HelloRequestList(com.hazelcast.jet.grpc.greeter.GreeterOuterClass.HelloRequestList) Server(io.grpc.Server) ExceptionUtil(com.hazelcast.jet.impl.util.ExceptionUtil) SimpleTestInClusterSupport(com.hazelcast.jet.SimpleTestInClusterSupport) GrpcServices.unaryService(com.hazelcast.jet.grpc.GrpcServices.unaryService) Pipeline(com.hazelcast.jet.pipeline.Pipeline) HelloReplyList(com.hazelcast.jet.grpc.greeter.GreeterOuterClass.HelloReplyList) HelloRequest(com.hazelcast.jet.grpc.greeter.GreeterOuterClass.HelloRequest) Sinks(com.hazelcast.jet.pipeline.Sinks) Test(org.junit.Test) IOException(java.io.IOException) GrpcServices.bidirectionalStreamingService(com.hazelcast.jet.grpc.GrpcServices.bidirectionalStreamingService) Category(org.junit.experimental.categories.Category) Executors(java.util.concurrent.Executors) TimeUnit(java.util.concurrent.TimeUnit) ManagedChannelBuilder(io.grpc.ManagedChannelBuilder) TestSources(com.hazelcast.jet.pipeline.test.TestSources) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) BatchStageWithKey(com.hazelcast.jet.pipeline.BatchStageWithKey) Assert.assertEquals(org.junit.Assert.assertEquals) HelloRequest(com.hazelcast.jet.grpc.greeter.GreeterOuterClass.HelloRequest) Pipeline(com.hazelcast.jet.pipeline.Pipeline) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Example 12 with BatchStage

use of com.hazelcast.jet.pipeline.BatchStage in project hazelcast-jet by hazelcast.

the class GrAggBuilder method buildBatch.

@SuppressWarnings("unchecked")
public <A, R, OUT> BatchStage<OUT> buildBatch(@Nonnull AggregateOperation<A, ? extends R> aggrOp, @Nonnull DistributedBiFunction<? super K, ? super R, OUT> mapToOutputFn) {
    List<Transform> upstreamTransforms = upstreamStages.stream().map(s -> s.transform).collect(toList());
    Transform transform = new GroupTransform<>(upstreamTransforms, keyFns, aggrOp, mapToOutputFn);
    pipelineImpl.connect(upstreamTransforms, transform);
    return new BatchStageImpl<>(transform, pipelineImpl);
}
Also used : WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) DistributedBiFunction(com.hazelcast.jet.function.DistributedBiFunction) ArrayList(java.util.ArrayList) BatchStage(com.hazelcast.jet.pipeline.BatchStage) Tag.tag(com.hazelcast.jet.datamodel.Tag.tag) KeyedWindowResultFunction(com.hazelcast.jet.function.KeyedWindowResultFunction) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) WindowGroupAggregateBuilder(com.hazelcast.jet.pipeline.WindowGroupAggregateBuilder) StageWithGroupingAndWindow(com.hazelcast.jet.pipeline.StageWithGroupingAndWindow) DistributedFunction(com.hazelcast.jet.function.DistributedFunction) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) Nonnull(javax.annotation.Nonnull) StreamStage(com.hazelcast.jet.pipeline.StreamStage) WindowDefinition(com.hazelcast.jet.pipeline.WindowDefinition) StageWithGrouping(com.hazelcast.jet.pipeline.StageWithGrouping) JetEventFunctionAdapter.adaptAggregateOperation(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptAggregateOperation) Tag(com.hazelcast.jet.datamodel.Tag) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) ADAPT_TO_JET_EVENT(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ADAPT_TO_JET_EVENT) List(java.util.List) ComputeStageImplBase.ensureJetEvents(com.hazelcast.jet.impl.pipeline.ComputeStageImplBase.ensureJetEvents) Collectors.toList(java.util.stream.Collectors.toList) JetEventFunctionAdapter.adaptKeyFn(com.hazelcast.jet.impl.pipeline.JetEventFunctionAdapter.adaptKeyFn) StreamStageWithGrouping(com.hazelcast.jet.pipeline.StreamStageWithGrouping) GroupAggregateBuilder(com.hazelcast.jet.pipeline.GroupAggregateBuilder) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) WindowGroupTransform(com.hazelcast.jet.impl.pipeline.transform.WindowGroupTransform) GroupTransform(com.hazelcast.jet.impl.pipeline.transform.GroupTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform)

Aggregations

BatchStage (com.hazelcast.jet.pipeline.BatchStage)12 Pipeline (com.hazelcast.jet.pipeline.Pipeline)11 List (java.util.List)11 Sinks (com.hazelcast.jet.pipeline.Sinks)10 Test (org.junit.Test)10 Category (org.junit.experimental.categories.Category)10 TestSources (com.hazelcast.jet.pipeline.test.TestSources)9 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)9 QuickTest (com.hazelcast.test.annotation.QuickTest)9 Collectors.toList (java.util.stream.Collectors.toList)9 After (org.junit.After)9 Assert.fail (org.junit.Assert.fail)9 SimpleTestInClusterSupport (com.hazelcast.jet.SimpleTestInClusterSupport)8 AssertionSinks (com.hazelcast.jet.pipeline.test.AssertionSinks)8 IOException (java.io.IOException)8 IntStream (java.util.stream.IntStream)8 Assert.assertEquals (org.junit.Assert.assertEquals)8 BeforeClass (org.junit.BeforeClass)8 Arrays (java.util.Arrays)7 GrpcServices.bidirectionalStreamingService (com.hazelcast.jet.grpc.GrpcServices.bidirectionalStreamingService)6