Search in sources :

Example 11 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachMapUsingPartitionedServiceAsync.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, K, R, RET> RET attachMapUsingPartitionedServiceAsync(@Nonnull ServiceFactory<?, S> serviceFactory, int maxConcurrentOps, boolean preserveOrder, @Nonnull FunctionEx<? super T, ? extends K> partitionKeyFn, @Nonnull BiFunctionEx<? super S, ? super T, ? extends CompletableFuture<R>> mapAsyncFn) {
    checkSerializable(mapAsyncFn, "mapAsyncFn");
    checkSerializable(partitionKeyFn, "partitionKeyFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx<? super S, ? super T, ? extends CompletableFuture<Traverser<R>>> flatMapAsyncFn = (s, t) -> mapAsyncFn.apply(s, t).thenApply(Traversers::singleton);
    BiFunctionEx adaptedFlatMapFn = fnAdapter.adaptFlatMapUsingServiceAsyncFn(flatMapAsyncFn);
    FunctionEx adaptedPartitionKeyFn = fnAdapter.adaptKeyFn(partitionKeyFn);
    PartitionedProcessorTransform processorTransform = flatMapUsingServiceAsyncPartitionedTransform(transform, "map", serviceFactory, maxConcurrentOps, preserveOrder, adaptedFlatMapFn, adaptedPartitionKeyFn);
    return attach(processorTransform, fnAdapter);
}
Also used : Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform.partitionedCustomProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.partitionedCustomProcessorTransform) PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncBatchedPartitionedTransform) ProcessorTransform.flatMapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceTransform) ProcessorTransform.mapUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.mapUsingServiceTransform) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) JoinClause(com.hazelcast.jet.pipeline.JoinClause) Traversers.traverseIterable(com.hazelcast.jet.Traversers.traverseIterable) Arrays.asList(java.util.Arrays.asList) ProcessorTransform.flatMapUsingServiceAsyncTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncTransform) Preconditions.checkTrue(com.hazelcast.internal.util.Preconditions.checkTrue) PredicateEx(com.hazelcast.function.PredicateEx) GlobalMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalMapStatefulTransform) PeekTransform(com.hazelcast.jet.impl.pipeline.transform.PeekTransform) GeneralStage(com.hazelcast.jet.pipeline.GeneralStage) FunctionEx(com.hazelcast.function.FunctionEx) PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServiceAsyncPartitionedTransform) Collections.emptyList(java.util.Collections.emptyList) SupplierEx(com.hazelcast.function.SupplierEx) List(java.util.List) JetEvent.jetEvent(com.hazelcast.jet.impl.JetEvent.jetEvent) ProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform) MergeTransform(com.hazelcast.jet.impl.pipeline.transform.MergeTransform) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) Sink(com.hazelcast.jet.pipeline.Sink) MapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.MapStatefulTransform) ComparatorEx(com.hazelcast.function.ComparatorEx) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) JetException(com.hazelcast.jet.JetException) BatchStage(com.hazelcast.jet.pipeline.BatchStage) GlobalFlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.GlobalFlatMapStatefulTransform) SinkTransform(com.hazelcast.jet.impl.pipeline.transform.SinkTransform) ProcessorTransform.filterUsingServiceTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.filterUsingServiceTransform) PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.flatMapUsingServicePartitionedTransform) DEFAULT_IDLE_TIMEOUT(com.hazelcast.jet.core.EventTimePolicy.DEFAULT_IDLE_TIMEOUT) MapTransform(com.hazelcast.jet.impl.pipeline.transform.MapTransform) TimestampTransform(com.hazelcast.jet.impl.pipeline.transform.TimestampTransform) ServiceFactory(com.hazelcast.jet.pipeline.ServiceFactory) Nonnull(javax.annotation.Nonnull) Nullable(javax.annotation.Nullable) BiPredicateEx(com.hazelcast.function.BiPredicateEx) FlatMapStatefulTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapStatefulTransform) StreamStage(com.hazelcast.jet.pipeline.StreamStage) ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.flatMapUsingServiceAsyncBatchedTransform) Util.toList(com.hazelcast.jet.impl.util.Util.toList) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) PartitionedProcessorTransform.filterUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.filterUsingServicePartitionedTransform) WatermarkPolicy.limitingLag(com.hazelcast.jet.core.WatermarkPolicy.limitingLag) Traversers(com.hazelcast.jet.Traversers) ProcessorTransform.customProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.ProcessorTransform.customProcessorTransform) AbstractTransform(com.hazelcast.jet.impl.pipeline.transform.AbstractTransform) Transform(com.hazelcast.jet.impl.pipeline.transform.Transform) PartitionedProcessorTransform.mapUsingServicePartitionedTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform.mapUsingServicePartitionedTransform) SinkStage(com.hazelcast.jet.pipeline.SinkStage) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) HashJoinTransform(com.hazelcast.jet.impl.pipeline.transform.HashJoinTransform) SortTransform(com.hazelcast.jet.impl.pipeline.transform.SortTransform) EventTimePolicy.eventTimePolicy(com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy) FlatMapTransform(com.hazelcast.jet.impl.pipeline.transform.FlatMapTransform) TriFunction(com.hazelcast.jet.function.TriFunction) Traverser(com.hazelcast.jet.Traverser) PartitionedProcessorTransform(com.hazelcast.jet.impl.pipeline.transform.PartitionedProcessorTransform) BiFunctionEx(com.hazelcast.function.BiFunctionEx) FunctionEx(com.hazelcast.function.FunctionEx) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) Traversers(com.hazelcast.jet.Traversers) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 12 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class ComputeStageImplBase method attachFlatMapUsingPartitionedService.

@Nonnull
@SuppressWarnings({ "unchecked", "rawtypes" })
<S, K, R, RET> RET attachFlatMapUsingPartitionedService(@Nonnull ServiceFactory<?, S> serviceFactory, @Nonnull FunctionEx<? super T, ? extends K> partitionKeyFn, @Nonnull BiFunctionEx<? super S, ? super T, ? extends Traverser<R>> flatMapFn) {
    checkSerializable(flatMapFn, "flatMapFn");
    checkSerializable(partitionKeyFn, "partitionKeyFn");
    serviceFactory = moveAttachedFilesToPipeline(serviceFactory);
    BiFunctionEx adaptedFlatMapFn = fnAdapter.adaptFlatMapUsingServiceFn(flatMapFn);
    FunctionEx adaptedPartitionKeyFn = fnAdapter.adaptKeyFn(partitionKeyFn);
    return (RET) attach(flatMapUsingServicePartitionedTransform(transform, serviceFactory, adaptedFlatMapFn, adaptedPartitionKeyFn), fnAdapter);
}
Also used : BiFunctionEx(com.hazelcast.function.BiFunctionEx) FunctionEx(com.hazelcast.function.FunctionEx) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) BiFunctionEx(com.hazelcast.function.BiFunctionEx) Nonnull(javax.annotation.Nonnull)

Example 13 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class HashJoinTransform method addToDag.

// ---------           ----------           ----------
// | primary |         | joined-1 |         | joined-2 |
// ---------           ----------           ----------
// |                   |                     |
// |              distributed          distributed
// |               broadcast            broadcast
// |                   v                     v
// |             -------------         -------------
// |            | collector-1 |       | collector-2 |
// |            | localPara=1 |       | localPara=1 |
// |             -------------         -------------
// |                   |                     |
// |                 local                 local
// local             broadcast             broadcast
// unicast           prioritized           prioritized
// ordinal 0           ordinal 1             ordinal 2
// \                   |                     |
// ----------------\  |   /----------------/
// v  v  v
// --------
// | joiner |
// --------
@Override
@SuppressWarnings("unchecked")
public void addToDag(Planner p, Context context) {
    determineLocalParallelism(LOCAL_PARALLELISM_USE_DEFAULT, context, p.isPreserveOrder());
    PlannerVertex primary = p.xform2vertex.get(this.upstream().get(0));
    List keyFns = toList(this.clauses, JoinClause::leftKeyFn);
    List<Tag> tags = this.tags;
    BiFunctionEx mapToOutputBiFn = this.mapToOutputBiFn;
    TriFunction mapToOutputTriFn = this.mapToOutputTriFn;
    // must be extracted to variable, probably because of serialization bug
    BiFunctionEx<List<Tag>, Object[], ItemsByTag> tupleToItems = tupleToItemsByTag(whereNullsNotAllowed);
    Vertex joiner = p.addVertex(this, name() + "-joiner", determinedLocalParallelism(), () -> new HashJoinP<>(keyFns, tags, mapToOutputBiFn, mapToOutputTriFn, tupleToItems)).v;
    Edge edgeToJoiner = from(primary.v, primary.nextAvailableOrdinal()).to(joiner, 0);
    if (p.isPreserveOrder()) {
        edgeToJoiner.isolated();
    } else {
        applyRebalancing(edgeToJoiner, this);
    }
    p.dag.edge(edgeToJoiner);
    String collectorName = name() + "-collector";
    int collectorOrdinal = 1;
    for (Transform fromTransform : tailList(this.upstream())) {
        PlannerVertex fromPv = p.xform2vertex.get(fromTransform);
        JoinClause<?, ?, ?, ?> clause = this.clauses.get(collectorOrdinal - 1);
        FunctionEx<Object, Object> getKeyFn = (FunctionEx<Object, Object>) clause.rightKeyFn();
        FunctionEx<Object, Object> projectFn = (FunctionEx<Object, Object>) clause.rightProjectFn();
        Vertex collector = p.dag.newVertex(collectorName + collectorOrdinal, () -> new HashJoinCollectP(getKeyFn, projectFn));
        collector.localParallelism(1);
        p.dag.edge(from(fromPv.v, fromPv.nextAvailableOrdinal()).to(collector, 0).distributed().broadcast());
        p.dag.edge(from(collector, 0).to(joiner, collectorOrdinal).broadcast().priority(-1));
        collectorOrdinal++;
    }
}
Also used : Vertex(com.hazelcast.jet.core.Vertex) PlannerVertex(com.hazelcast.jet.impl.pipeline.Planner.PlannerVertex) HashJoinCollectP(com.hazelcast.jet.impl.processor.HashJoinCollectP) HashJoinP(com.hazelcast.jet.impl.processor.HashJoinP) ItemsByTag(com.hazelcast.jet.datamodel.ItemsByTag) PlannerVertex(com.hazelcast.jet.impl.pipeline.Planner.PlannerVertex) FunctionEx(com.hazelcast.function.FunctionEx) BiFunctionEx(com.hazelcast.function.BiFunctionEx) TriFunction(com.hazelcast.jet.function.TriFunction) Util.toList(com.hazelcast.jet.impl.util.Util.toList) List(java.util.List) Planner.tailList(com.hazelcast.jet.impl.pipeline.Planner.tailList) Tag(com.hazelcast.jet.datamodel.Tag) ItemsByTag(com.hazelcast.jet.datamodel.ItemsByTag) JoinClause(com.hazelcast.jet.pipeline.JoinClause) Edge(com.hazelcast.jet.core.Edge) BiFunctionEx(com.hazelcast.function.BiFunctionEx)

Example 14 with BiFunctionEx

use of com.hazelcast.function.BiFunctionEx in project hazelcast by hazelcast.

the class CreateDagVisitor method onSlidingWindow.

public Vertex onSlidingWindow(SlidingWindowPhysicalRel rel) {
    int orderingFieldIndex = rel.orderingFieldIndex();
    FunctionEx<ExpressionEvalContext, SlidingWindowPolicy> windowPolicySupplier = rel.windowPolicyProvider();
    // this vertex is used only if there's no aggregation by a window bound
    Vertex vertex = dag.newUniqueVertex("Sliding-Window", flatMapUsingServiceP(ServiceFactories.nonSharedService(ctx -> {
        ExpressionEvalContext evalContext = ExpressionEvalContext.from(ctx);
        SlidingWindowPolicy windowPolicy = windowPolicySupplier.apply(evalContext);
        return row -> WindowUtils.addWindowBounds(row, orderingFieldIndex, windowPolicy);
    }), (BiFunctionEx<Function<JetSqlRow, Traverser<JetSqlRow>>, JetSqlRow, Traverser<JetSqlRow>>) Function::apply));
    connectInput(rel.getInput(), vertex, null);
    return vertex;
}
Also used : ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Address(com.hazelcast.cluster.Address) Traverser(com.hazelcast.jet.Traverser) ExpressionValues(com.hazelcast.jet.sql.impl.opt.ExpressionValues) PlanObjectKey(com.hazelcast.sql.impl.optimizer.PlanObjectKey) Processors.mapP(com.hazelcast.jet.core.processor.Processors.mapP) SourceProcessors.convenientSourceP(com.hazelcast.jet.core.processor.SourceProcessors.convenientSourceP) QueryParameterMetadata(com.hazelcast.sql.impl.QueryParameterMetadata) Collections.singletonList(java.util.Collections.singletonList) BiFunctionEx(com.hazelcast.function.BiFunctionEx) HazelcastTable(com.hazelcast.jet.sql.impl.schema.HazelcastTable) ObjectArrayKey(com.hazelcast.jet.sql.impl.ObjectArrayKey) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) SqlConnectorUtil(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Functions.entryKey(com.hazelcast.function.Functions.entryKey) SqlConnectorUtil.getJetSqlConnector(com.hazelcast.jet.sql.impl.connector.SqlConnectorUtil.getJetSqlConnector) DAG(com.hazelcast.jet.core.DAG) RootResultConsumerSink.rootResultConsumerSink(com.hazelcast.jet.sql.impl.processors.RootResultConsumerSink.rootResultConsumerSink) ServiceFactories(com.hazelcast.jet.pipeline.ServiceFactories) FunctionEx(com.hazelcast.function.FunctionEx) KeyedWindowResultFunction(com.hazelcast.jet.core.function.KeyedWindowResultFunction) Predicate(java.util.function.Predicate) Collections.emptyList(java.util.Collections.emptyList) Set(java.util.Set) ConsumerEx(com.hazelcast.function.ConsumerEx) IMapSqlConnector(com.hazelcast.jet.sql.impl.connector.map.IMapSqlConnector) ExpressionUtil(com.hazelcast.jet.sql.impl.ExpressionUtil) Processors.filterUsingServiceP(com.hazelcast.jet.core.processor.Processors.filterUsingServiceP) List(java.util.List) ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Processors.mapUsingServiceP(com.hazelcast.jet.core.processor.Processors.mapUsingServiceP) Table(com.hazelcast.sql.impl.schema.Table) ComparatorEx(com.hazelcast.function.ComparatorEx) QueryDataType(com.hazelcast.sql.impl.type.QueryDataType) Processors(com.hazelcast.jet.core.processor.Processors) Processors.flatMapUsingServiceP(com.hazelcast.jet.core.processor.Processors.flatMapUsingServiceP) TimestampKind(com.hazelcast.jet.core.TimestampKind) Function(java.util.function.Function) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) HashSet(java.util.HashSet) Edge.from(com.hazelcast.jet.core.Edge.from) Edge(com.hazelcast.jet.core.Edge) Expression(com.hazelcast.sql.impl.expression.Expression) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) VertexWithInputConfig(com.hazelcast.jet.sql.impl.connector.SqlConnector.VertexWithInputConfig) SingleRel(org.apache.calcite.rel.SingleRel) Nullable(javax.annotation.Nullable) JetJoinInfo(com.hazelcast.jet.sql.impl.JetJoinInfo) NodeEngine(com.hazelcast.spi.impl.NodeEngine) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) RelNode(org.apache.calcite.rel.RelNode) Consumer(java.util.function.Consumer) Vertex(com.hazelcast.jet.core.Vertex) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) SqlHashJoinP(com.hazelcast.jet.sql.impl.processors.SqlHashJoinP) ConstantExpression(com.hazelcast.sql.impl.expression.ConstantExpression) Processors.sortP(com.hazelcast.jet.core.processor.Processors.sortP) DefaultSerializationServiceBuilder(com.hazelcast.internal.serialization.impl.DefaultSerializationServiceBuilder) WindowUtils(com.hazelcast.jet.sql.impl.aggregate.WindowUtils) Edge.between(com.hazelcast.jet.core.Edge.between) Vertex(com.hazelcast.jet.core.Vertex) KeyedWindowResultFunction(com.hazelcast.jet.core.function.KeyedWindowResultFunction) Function(java.util.function.Function) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Traverser(com.hazelcast.jet.Traverser) JetSqlRow(com.hazelcast.sql.impl.row.JetSqlRow) BiFunctionEx(com.hazelcast.function.BiFunctionEx)

Aggregations

BiFunctionEx (com.hazelcast.function.BiFunctionEx)14 FunctionEx (com.hazelcast.function.FunctionEx)10 Nonnull (javax.annotation.Nonnull)10 List (java.util.List)8 Traverser (com.hazelcast.jet.Traverser)7 ToLongFunctionEx (com.hazelcast.function.ToLongFunctionEx)6 TriFunction (com.hazelcast.jet.function.TriFunction)6 Function (java.util.function.Function)6 EventTimePolicy.eventTimePolicy (com.hazelcast.jet.core.EventTimePolicy.eventTimePolicy)5 Util.toList (com.hazelcast.jet.impl.util.Util.toList)5 ComparatorEx (com.hazelcast.function.ComparatorEx)4 SupplierEx (com.hazelcast.function.SupplierEx)4 Traversers.traverseIterable (com.hazelcast.jet.Traversers.traverseIterable)4 ProcessorMetaSupplier (com.hazelcast.jet.core.ProcessorMetaSupplier)4 ServiceFactory (com.hazelcast.jet.pipeline.ServiceFactory)4 CompletableFuture (java.util.concurrent.CompletableFuture)4 BiPredicateEx (com.hazelcast.function.BiPredicateEx)3 PredicateEx (com.hazelcast.function.PredicateEx)3 Preconditions.checkTrue (com.hazelcast.internal.util.Preconditions.checkTrue)3 JetException (com.hazelcast.jet.JetException)3