Examples with ProcessorMetaSupplier com.hazelcast.jet.core.ProcessorMetaSupplier used on opensource projects

Example 11 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class SinksTest method mapWithMerging_when_multipleValuesForSingleKeyInABatch.

@Test
public void mapWithMerging_when_multipleValuesForSingleKeyInABatch() throws Exception {
    ProcessorMetaSupplier metaSupplier = adaptSupplier(SinkProcessors.<Entry<String, Integer>, String, Integer>mergeMapP(sinkName, Entry::getKey, Entry::getValue, Integer::sum));
    TestProcessorSupplierContext psContext = new TestProcessorSupplierContext().setHazelcastInstance(member);
    Processor p = TestSupport.supplierFrom(metaSupplier, psContext).get();
    TestOutbox outbox = new TestOutbox();
    p.init(outbox, new TestProcessorContext().setHazelcastInstance(member));
    TestInbox inbox = new TestInbox();
    inbox.add(entry("k", 1));
    inbox.add(entry("k", 2));
    p.process(0, inbox);
    assertTrue("inbox.isEmpty()", inbox.isEmpty());
    assertTrueEventually(() -> {
        assertTrue("p.complete()", p.complete());
    }, 10);
    p.close();
    // assert the output map contents
    IMap<Object, Object> actual = member.getMap(sinkName);
    assertEquals(1, actual.size());
    assertEquals(3, actual.get("k"));
}

Example 12 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class StreamJmsPTest method when_sharedConsumer_then_twoProcessorsUsed.

@Test
public void when_sharedConsumer_then_twoProcessorsUsed() throws Exception {
    String topicName = randomString();
    logger.info("using topic: " + topicName);
    StreamSource<Message> source = Sources.jmsTopicBuilder(StreamJmsPTest::getConnectionFactory).destinationName("foo").sharedConsumer(true).build();
    ProcessorMetaSupplier metaSupplier = ((StreamSourceTransform<Message>) source).metaSupplierFn.apply(noEventTime());
    Address address1 = new Address("127.0.0.1", 1);
    Address address2 = new Address("127.0.0.1", 2);
    Function<? super Address, ? extends ProcessorSupplier> function = metaSupplier.get(asList(address1, address2));
    // Then
    // assert that processors for both addresses are actually StreamJmsP, not a noopP
    assertInstanceOf(StreamJmsP.class, function.apply(address1).get(1).iterator().next());
    assertInstanceOf(StreamJmsP.class, function.apply(address2).get(1).iterator().next());
}

Example 13 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class StreamFilesPTest method when_metaSupplier_then_returnsCorrectProcessors.

@Test
public void when_metaSupplier_then_returnsCorrectProcessors() throws Exception {
    ProcessorMetaSupplier metaSupplier = streamFilesP(workDir.getAbsolutePath(), UTF_8, "*", false, Util::entry);
    Address a = new Address();
    ProcessorSupplier supplier = metaSupplier.get(singletonList(a)).apply(a);
    supplier.init(new TestProcessorContext());
    assertEquals(1, supplier.get(1).size());
    supplier.close(null);
}

Example 14 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class PartitionedProcessorTransform method flatMapUsingServiceAsyncPartitionedTransform.

public static <S, T, K, R> PartitionedProcessorTransform<T, K> flatMapUsingServiceAsyncPartitionedTransform(@Nonnull Transform upstream, @Nonnull String operationName, @Nonnull ServiceFactory<?, S> serviceFactory, int maxConcurrentOps, boolean preserveOrder, @Nonnull BiFunctionEx<? super S, ? super T, CompletableFuture<Traverser<R>>> flatMapAsyncFn, @Nonnull FunctionEx<? super T, ? extends K> partitionKeyFn) {
    String name = operationName + "UsingPartitionedServiceAsync";
    ProcessorSupplier supplier = flatMapUsingServiceAsyncP(serviceFactory, maxConcurrentOps, preserveOrder, partitionKeyFn, flatMapAsyncFn);
    ProcessorMetaSupplier metaSupplier = ProcessorMetaSupplier.of(getPreferredLP(serviceFactory), serviceFactory.permission(), supplier);
    return new PartitionedProcessorTransform<>(name, upstream, metaSupplier, partitionKeyFn);
}

Example 15 with ProcessorMetaSupplier

use of com.hazelcast.jet.core.ProcessorMetaSupplier in project hazelcast by hazelcast.

the class ProcessorTransform method flatMapUsingServiceAsyncTransform.

public static <S, T, R> ProcessorTransform flatMapUsingServiceAsyncTransform(@Nonnull Transform upstream, @Nonnull String operationName, @Nonnull ServiceFactory<?, S> serviceFactory, int maxConcurrentOps, boolean preserveOrder, @Nonnull BiFunctionEx<? super S, ? super T, ? extends CompletableFuture<Traverser<R>>> flatMapAsyncFn) {
    // TODO use better key so that snapshots are local. Currently they will
    // be sent to a random member. We keep it this way for simplicity:
    // the number of in-flight items is limited (maxConcurrentOps)
    ProcessorSupplier supplier = flatMapUsingServiceAsyncP(serviceFactory, maxConcurrentOps, preserveOrder, Object::hashCode, flatMapAsyncFn);
    ProcessorMetaSupplier metaSupplier = ProcessorMetaSupplier.of(getPreferredLP(serviceFactory), serviceFactory.permission(), supplier);
    return new ProcessorTransform(operationName + "UsingServiceAsync", upstream, metaSupplier);
}