Search in sources :

Example 1 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingServiceAsync.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, R, RET> RET attachMapUsingServiceAsync(@Nonnull ServiceFactory<?, S> serviceFactory, int maxConcurrentOps, boolean preserveOrder, @Nonnull BiFunctionEx<? super S, ? super T, ? extends CompletableFuture<Traverser<R>>> flatMapAsyncFn) {
    checkSerializable(flatMapAsyncFn, "mapAsyncFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx adaptedFlatMapFn = fnAdapter.adaptFlatMapUsingServiceAsyncFn(flatMapAsyncFn);
    ProcessorTransform processorTransform = flatMapUsingServiceAsyncTransform(transform, "map", serviceFactory, maxConcurrentOps, preserveOrder, adaptedFlatMapFn);
    return attach(processorTransform, fnAdapter);
}
Also used : PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 2 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingPartitionedService.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, K, R, RET> RET attachMapUsingPartitionedService(@Nonnull ServiceFactory<?, S> serviceFactory, @Nonnull FunctionEx<? super T, ? extends K> partitionKeyFn, @Nonnull BiFunctionEx<? super S, ? super T, ? extends R> mapFn) {
    checkSerializable(mapFn, "mapFn");
    checkSerializable(partitionKeyFn, "partitionKeyFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx adaptedMapFn = fnAdapter.adaptMapUsingServiceFn(mapFn);
    FunctionEx adaptedPartitionKeyFn = fnAdapter.adaptKeyFn(partitionKeyFn);
    return (RET) attach(mapUsingServicePartitionedTransform(transform, serviceFactory, adaptedMapFn, adaptedPartitionKeyFn), fnAdapter);
}
Also used : BiFunctionEx(com.hazelcast.function.BiFunctionEx) FunctionEx(com.hazelcast.function.FunctionEx) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 3 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingService.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, R, RET> RET attachMapUsingService(@Nonnull ServiceFactory<?, S> serviceFactory, @Nonnull BiFunctionEx<? super S, ? super T, ? extends R> mapFn) {
    checkSerializable(mapFn, "mapFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx adaptedMapFn = fnAdapter.adaptMapUsingServiceFn(mapFn);
    return attach(mapUsingServiceTransform(transform, serviceFactory, adaptedMapFn), fnAdapter);
}
Also used : BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 4 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingPartitionedServiceAsyncBatched.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, K, R, RET> RET attachMapUsingPartitionedServiceAsyncBatched(@Nonnull ServiceFactory<?, S> serviceFactory, int maxBatchSize, @Nonnull FunctionEx<? super T, ? extends K> partitionKeyFn, @Nonnull BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<R>>> mapAsyncFn) {
    checkSerializable(mapAsyncFn, "mapAsyncFn");
    checkSerializable(partitionKeyFn, "partitionKeyFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<Traverser<R>>>> flatMapAsyncFn = (s, items) -> mapAsyncFn.apply(s, items).thenApply(list -> toList(list, Traversers::singleton));
    BiFunctionEx adaptedFlatMapFn = fnAdapter.adaptFlatMapUsingServiceAsyncBatchedFn(flatMapAsyncFn);
    FunctionEx adaptedPartitionKeyFn = fnAdapter.adaptKeyFn(partitionKeyFn);
    // Here we flatten the result from List<Traverser<R>> to Traverser<R>.
    // The former is used in pipeline API, the latter in core API.
    BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<Traverser<R>>> flattenedFn = (svc, items) -> {
        // R might actually be JetEvent<R> -- we can't represent this with static types
        CompletableFuture<List<Traverser<R>>> f = (CompletableFuture<List<Traverser<R>>>) adaptedFlatMapFn.apply(svc, items);
        return f.thenApply(res -> traverseIterable(res).flatMap(Function.identity()));
    };
    PartitionedProcessorTransform processorTransform = flatMapUsingServiceAsyncBatchedPartitionedTransform(transform, "map", serviceFactory, MAX_CONCURRENT_ASYNC_BATCHES, maxBatchSize, flattenedFn, adaptedPartitionKeyFn);
    return attach(processorTransform, fnAdapter);
}
Also used : Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform) ProcessorTransform.flatMapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceTransform) ProcessorTransform.mapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.mapUsingServiceTransform) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) JoinClause(com.hazelcast.jet.pipeline.JoinClause) Traversers.traverseIterable(com.hazelcast.jet.Traversers.traverseIterable) Arrays.asList(java.util.Arrays.asList) ProcessorTransform.flatMapUsingServiceAsyncTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncTransform) Preconditions.checkTrue(com.hazelcast.internal.util.Preconditions.checkTrue) PredicateEx(com.hazelcast.function.PredicateEx) GlobalMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalMapStatefulTransform) PeekTransform(com.hazelcast.jet.impl.pipeline.transform.PeekTransform) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) FunctionEx(com.hazelcast.function.FunctionEx) PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform) Collections.emptyList(java.util.Collections.emptyList) SupplierEx(com.hazelcast.function.SupplierEx) List(java.util.List) JetEvent.jetEvent(com.hazelcast.jet.impl.JetEvent.jetEvent) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) MergeTransform(com.hazelcast.jet.impl.pipeline.transform.MergeTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) Sink(com.hazelcast.jet.pipeline.Sink) MapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.MapStatefulTransform) ComparatorEx(com.hazelcast.function.ComparatorEx) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JetException(com.hazelcast.jet.JetException) BatchStage(com.hazelcast.jet.pipeline.BatchStage) GlobalFlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalFlatMapStatefulTransform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) ProcessorTransform.filterUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.filterUsingServiceTransform) PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) MapTransform(com.hazelcast.jet.impl.pipeline.transform.MapTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) ServiceFactory(com.hazelcast.jet.pipeline.ServiceFactory) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) BiPredicateEx(com.hazelcast.function.BiPredicateEx) FlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapStatefulTransform) StreamStage(com.hazelcast.jet.pipeline.StreamStage) ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform) Util.toList(com.hazelcast.jet.impl.util.Util.toList) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) PartitionedProcessorTransform.filterUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.filterUsingServicePartitionedTransform) WatermarkPolicy.limitingLag(com.hazelcast.jet.core.WatermarkPolicy.limitingLag) Traversers(com.hazelcast.jet.Traversers) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) PartitionedProcessorTransform.mapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.mapUsingServicePartitionedTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) HashJoinTransform(com.hazelcast.jet.impl.pipeline.transform.HashJoinTransform) SortTransform(com.hazelcast.jet.impl.pipeline.transform.SortTransform) EventTimePolicy.eventTimePolicy(com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy) FlatMapTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform) TriFunction(com.hazelcast.jet.function.TriFunction) CompletableFuture(java.util.concurrent.CompletableFuture) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) BiFunctionEx(com.hazelcast.function.BiFunctionEx) FunctionEx(com.hazelcast.function.FunctionEx) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) Collections.emptyList(java.util.Collections.emptyList) List(java.util.List) Util.toList(com.hazelcast.jet.impl.util.Util.toList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 5 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingServiceAsyncBatched.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, R, RET> RET attachMapUsingServiceAsyncBatched(@Nonnull ServiceFactory<?, S> serviceFactory, int maxBatchSize, @Nonnull BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<List<Traverser<R>>>> flatMapAsyncBatchedFn) {
    checkSerializable(flatMapAsyncBatchedFn, "mapAsyncBatchedFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx adaptedFn = fnAdapter.adaptFlatMapUsingServiceAsyncBatchedFn(flatMapAsyncBatchedFn);
    // Here we flatten the result from List<Traverser<R>> to Traverser<R>.
    // The former is used in pipeline API, the latter in core API.
    BiFunctionEx<? super S, ? super List<T>, ? extends CompletableFuture<Traverser<R>>> flattenedFn = (svc, items) -> {
        // R might actually be JetEvent<R> -- we can't represent this with static types
        CompletableFuture<List<Traverser<R>>> f = (CompletableFuture<List<Traverser<R>>>) adaptedFn.apply(svc, items);
        return f.thenApply(res -> traverseIterable(res).flatMap(Function.identity()));
    };
    ProcessorTransform processorTransform = flatMapUsingServiceAsyncBatchedTransform(transform, "map", serviceFactory, MAX_CONCURRENT_ASYNC_BATCHES, maxBatchSize, flattenedFn);
    return attach(processorTransform, fnAdapter);
}
Also used : Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform) ProcessorTransform.flatMapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceTransform) ProcessorTransform.mapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.mapUsingServiceTransform) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) JoinClause(com.hazelcast.jet.pipeline.JoinClause) Traversers.traverseIterable(com.hazelcast.jet.Traversers.traverseIterable) Arrays.asList(java.util.Arrays.asList) ProcessorTransform.flatMapUsingServiceAsyncTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncTransform) Preconditions.checkTrue(com.hazelcast.internal.util.Preconditions.checkTrue) PredicateEx(com.hazelcast.function.PredicateEx) GlobalMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalMapStatefulTransform) PeekTransform(com.hazelcast.jet.impl.pipeline.transform.PeekTransform) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) FunctionEx(com.hazelcast.function.FunctionEx) PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform) Collections.emptyList(java.util.Collections.emptyList) SupplierEx(com.hazelcast.function.SupplierEx) List(java.util.List) JetEvent.jetEvent(com.hazelcast.jet.impl.JetEvent.jetEvent) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) MergeTransform(com.hazelcast.jet.impl.pipeline.transform.MergeTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) Sink(com.hazelcast.jet.pipeline.Sink) MapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.MapStatefulTransform) ComparatorEx(com.hazelcast.function.ComparatorEx) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JetException(com.hazelcast.jet.JetException) BatchStage(com.hazelcast.jet.pipeline.BatchStage) GlobalFlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalFlatMapStatefulTransform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) ProcessorTransform.filterUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.filterUsingServiceTransform) PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) MapTransform(com.hazelcast.jet.impl.pipeline.transform.MapTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) ServiceFactory(com.hazelcast.jet.pipeline.ServiceFactory) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) BiPredicateEx(com.hazelcast.function.BiPredicateEx) FlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapStatefulTransform) StreamStage(com.hazelcast.jet.pipeline.StreamStage) ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform) Util.toList(com.hazelcast.jet.impl.util.Util.toList) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) PartitionedProcessorTransform.filterUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.filterUsingServicePartitionedTransform) WatermarkPolicy.limitingLag(com.hazelcast.jet.core.WatermarkPolicy.limitingLag) Traversers(com.hazelcast.jet.Traversers) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) PartitionedProcessorTransform.mapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.mapUsingServicePartitionedTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) HashJoinTransform(com.hazelcast.jet.impl.pipeline.transform.HashJoinTransform) SortTransform(com.hazelcast.jet.impl.pipeline.transform.SortTransform) EventTimePolicy.eventTimePolicy(com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy) FlatMapTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform) TriFunction(com.hazelcast.jet.function.TriFunction) CompletableFuture(java.util.concurrent.CompletableFuture) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) Traverser(com.hazelcast.jet.Traverser) Collections.singletonList(java.util.Collections.singletonList) Arrays.asList(java.util.Arrays.asList) Collections.emptyList(java.util.Collections.emptyList) List(java.util.List) Util.toList(com.hazelcast.jet.impl.util.Util.toList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Aggregations

BiFunctionEx (com.hazelcast.function.BiFunctionEx)14 FunctionEx (com.hazelcast.function.FunctionEx)10 Nonnull (javax.annotation.Nonnull)10 List (java.util.List)8 Traverser (com.hazelcast.jet.Traverser)7 ToLongFunctionEx (com.hazelcast.function.ToLongFunctionEx)6 TriFunction (com.hazelcast.jet.function.TriFunction)6 Function (java.util.function.Function)6 EventTimePolicy.eventTimePolicy (com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy)5 Util.toList (com.hazelcast.jet.impl.util.Util.toList)5 ComparatorEx (com.hazelcast.function.ComparatorEx)4 SupplierEx (com.hazelcast.function.SupplierEx)4 Traversers.traverseIterable (com.hazelcast.jet.Traversers.traverseIterable)4 ProcessorMetaSupplier (com.hazelcast.jet.core.ProcessorMetaSupplier)4 ServiceFactory (com.hazelcast.jet.pipeline.ServiceFactory)4 CompletableFuture (java.util.concurrent.CompletableFuture)4 BiPredicateEx (com.hazelcast.function.BiPredicateEx)3 PredicateEx (com.hazelcast.function.PredicateEx)3 Preconditions.checkTrue (com.hazelcast.internal.util.Preconditions.checkTrue)3 JetException (com.hazelcast.jet.JetException)3