Search in sources :

Example 1 with SlidingWindowPolicy

use of com.hazelcast.jet.core.SlidingWindowPolicy in project hazelcast-jet by hazelcast.

the class SlidingWindowPTest method before.

@Before
public void before() {
    SlidingWindowPolicy windowDef = slidingWinPolicy(4, 1);
    AggregateOperation1<Entry<?, Long>, LongAccumulator, Long> operation = AggregateOperation.withCreate(LongAccumulator::new).andAccumulate((LongAccumulator acc, Entry<?, Long> item) -> acc.add(item.getValue())).andCombine(LongAccumulator::add).andDeduct(hasDeduct ? LongAccumulator::subtract : null).andFinish(LongAccumulator::get);
    DistributedFunction<?, Long> keyFn = t -> KEY;
    DistributedToLongFunction<Entry<Long, Long>> timestampFn = Entry::getKey;
    DistributedSupplier<Processor> procSupplier = singleStageProcessor ? aggregateToSlidingWindowP(singletonList(keyFn), singletonList(timestampFn), TimestampKind.EVENT, windowDef, operation, TimestampedEntry::new) : combineToSlidingWindowP(windowDef, operation, TimestampedEntry::new);
    // new supplier to save the last supplied instance
    supplier = () -> lastSuppliedProcessor = (SlidingWindowP) procSupplier.get();
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Collections.shuffle(java.util.Collections.shuffle) Arrays(java.util.Arrays) DistributedToLongFunction(com.hazelcast.jet.function.DistributedToLongFunction) RunWith(org.junit.runner.RunWith) Parameters(org.junit.runners.Parameterized.Parameters) Processor(com.hazelcast.jet.core.Processor) TimestampKind(com.hazelcast.jet.core.TimestampKind) TestSupport.verifyProcessor(com.hazelcast.jet.core.test.TestSupport.verifyProcessor) SlidingWindowPolicy.slidingWinPolicy(com.hazelcast.jet.core.SlidingWindowPolicy.slidingWinPolicy) ArrayList(java.util.ArrayList) Collections.singletonList(java.util.Collections.singletonList) Watermark(com.hazelcast.jet.core.Watermark) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Processors.aggregateToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.aggregateToSlidingWindowP) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Util.entry(com.hazelcast.jet.Util.entry) Arrays.asList(java.util.Arrays.asList) After(org.junit.After) DistributedFunction(com.hazelcast.jet.function.DistributedFunction) Processors.combineToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP) ExpectedException(org.junit.rules.ExpectedException) Parameterized(org.junit.runners.Parameterized) Before(org.junit.Before) DistributedSupplier(com.hazelcast.jet.function.DistributedSupplier) LongStream(java.util.stream.LongStream) Parameter(org.junit.runners.Parameterized.Parameter) Collection(java.util.Collection) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) AggregateOperation1(com.hazelcast.jet.aggregate.AggregateOperation1) ParallelTest(com.hazelcast.test.annotation.ParallelTest) Category(org.junit.experimental.categories.Category) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) Rule(org.junit.Rule) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Entry(java.util.Map.Entry) HazelcastParametersRunnerFactory(com.hazelcast.test.HazelcastParametersRunnerFactory) TimestampedEntry(com.hazelcast.jet.datamodel.TimestampedEntry) Entry(java.util.Map.Entry) TimestampedEntry(com.hazelcast.jet.datamodel.TimestampedEntry) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Processor(com.hazelcast.jet.core.Processor) TestSupport.verifyProcessor(com.hazelcast.jet.core.test.TestSupport.verifyProcessor) Processors.aggregateToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.aggregateToSlidingWindowP) Processors.combineToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP) Before(org.junit.Before)

Example 2 with SlidingWindowPolicy

use of com.hazelcast.jet.core.SlidingWindowPolicy in project hazelcast-jet by hazelcast.

the class SlidingWindowP_twoStageSnapshotTest method before.

@Before
public void before() {
    SlidingWindowPolicy windowDef = slidingWinPolicy(4, 1);
    AggregateOperation1<Entry<?, Long>, LongAccumulator, Long> aggrOp = AggregateOperation.withCreate(LongAccumulator::new).andAccumulate((LongAccumulator acc, Entry<?, Long> item) -> acc.add(item.getValue())).andCombine(LongAccumulator::add).andDeduct(LongAccumulator::subtract).andFinish(LongAccumulator::get);
    DistributedSupplier<Processor> procSupplier1 = Processors.accumulateByFrameP(singletonList((DistributedFunction<? super Entry<Long, Long>, ?>) t -> KEY), singletonList((DistributedToLongFunction<? super Entry<Long, Long>>) Entry::getKey), TimestampKind.EVENT, windowDef, ((AggregateOperation1<? super Entry<Long, Long>, LongAccumulator, ?>) aggrOp).withFinishFn(identity()));
    DistributedSupplier<Processor> procSupplier2 = combineToSlidingWindowP(windowDef, aggrOp, TimestampedEntry::new);
    // new supplier to save the last supplied instance
    stage1Supplier = () -> lastSuppliedStage1Processor = (SlidingWindowP<?, ?, ?, ?>) procSupplier1.get();
    stage2Supplier = () -> lastSuppliedStage2Processor = (SlidingWindowP<?, ?, ?, ?>) procSupplier2.get();
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Entry(java.util.Map.Entry) TimestampedEntry(com.hazelcast.jet.datamodel.TimestampedEntry) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Processor(com.hazelcast.jet.core.Processor) TimestampedEntry(com.hazelcast.jet.datamodel.TimestampedEntry) DistributedToLongFunction(com.hazelcast.jet.function.DistributedToLongFunction) Processors.combineToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP) DistributedFunction(com.hazelcast.jet.function.DistributedFunction) Before(org.junit.Before)

Example 3 with SlidingWindowPolicy

use of com.hazelcast.jet.core.SlidingWindowPolicy in project hazelcast-jet by hazelcast.

the class WindowGroupTransform method addSlidingWindowTwoStage.

// ---------       ---------
// | source0 | ... | sourceN |
// ---------       ---------
// |               |
// local           local
// partitioned     partitioned
// v               v
// --------------------
// | accumulateByFrameP |
// --------------------
// |
// distributed
// partitioned
// v
// -------------------------
// | combineToSlidingWindowP |
// -------------------------
private void addSlidingWindowTwoStage(Planner p, SlidingWindowDef wDef) {
    String namePrefix = p.uniqueVertexName(name(), "-step");
    SlidingWindowPolicy winPolicy = wDef.toSlidingWindowPolicy();
    Vertex v1 = p.dag.newVertex(namePrefix + '1', accumulateByFrameP(keyFns, nCopies(keyFns.size(), (DistributedToLongFunction<JetEvent>) JetEvent::timestamp), TimestampKind.EVENT, winPolicy, aggrOp));
    v1.localParallelism(localParallelism());
    PlannerVertex pv2 = p.addVertex(this, namePrefix + '2', localParallelism(), combineToSlidingWindowP(winPolicy, aggrOp, mapToOutputFn));
    p.addEdges(this, v1, (e, ord) -> e.partitioned(keyFns.get(ord), HASH_CODE));
    p.dag.edge(between(v1, pv2.v).distributed().partitioned(entryKey()));
}
Also used : PlannerVertex(com.hazelcast.jet.impl.pipeline.Planner.PlannerVertex) Vertex(com.hazelcast.jet.core.Vertex) PlannerVertex(com.hazelcast.jet.impl.pipeline.Planner.PlannerVertex) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) JetEvent(com.hazelcast.jet.impl.pipeline.JetEvent)

Example 4 with SlidingWindowPolicy

use of com.hazelcast.jet.core.SlidingWindowPolicy in project hazelcast-jet-reference-manual by hazelcast.

the class StockExchangeCoreApi method buildDag.

static DAG buildDag() {
    // tag::s1[]
    DistributedToLongFunction<? super Trade> timestampFn = Trade::timestamp;
    DistributedFunction<? super Trade, ?> keyFn = Trade::productId;
    SlidingWindowPolicy winPolicy = slidingWinPolicy(SLIDING_WINDOW_LENGTH_MILLIS, SLIDE_STEP_MILLIS);
    DAG dag = new DAG();
    Vertex tradeSource = dag.newVertex("trade-source", SourceProcessors.<Trade, Long, Trade>streamMapP(TRADES_MAP_NAME, // <1>
    alwaysTrue(), // <1>
    EventJournalMapEvent::getNewValue, // <2>
    JournalInitialPosition.START_FROM_OLDEST, wmGenParams(// <3>
    timestampFn, // <4>
    limitingLag(SECONDS.toMillis(3)), // <5>
    emitByFrame(winPolicy), // <6>
    SECONDS.toMillis(3))));
    Vertex slidingStage1 = dag.newVertex("sliding-stage-1", Processors.accumulateByFrameP(singletonList(keyFn), singletonList(timestampFn), TimestampKind.EVENT, winPolicy, counting()));
    Vertex slidingStage2 = dag.newVertex("sliding-stage-2", Processors.combineToSlidingWindowP(winPolicy, counting(), TimestampedEntry::new));
    Vertex formatOutput = dag.newVertex("format-output", mapUsingContextP(ContextFactory.withCreateFn(x -> DateTimeFormatter.ofPattern("HH:mm:ss.SSS")), (DateTimeFormatter timeFormat, TimestampedEntry<String, Long> tse) -> String.format("%s %5s %4d", timeFormat.format(Instant.ofEpochMilli(tse.getTimestamp()).atZone(ZoneId.systemDefault())), tse.getKey(), tse.getValue())));
    Vertex sink = dag.newVertex("sink", SinkProcessors.writeFileP(OUTPUT_DIR_NAME));
    tradeSource.localParallelism(1);
    return dag.edge(between(tradeSource, slidingStage1).partitioned(keyFn, HASH_CODE)).edge(between(slidingStage1, slidingStage2).partitioned(entryKey(), HASH_CODE).distributed()).edge(between(slidingStage2, formatOutput).isolated()).edge(between(formatOutput, sink));
// end::s1[]
}
Also used : Vertex(com.hazelcast.jet.core.Vertex) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) DAG(com.hazelcast.jet.core.DAG) DateTimeFormatter(java.time.format.DateTimeFormatter)

Example 5 with SlidingWindowPolicy

use of com.hazelcast.jet.core.SlidingWindowPolicy in project hazelcast by hazelcast.

the class SlidingWindow method windowPolicyProvider.

public final FunctionEx<ExpressionEvalContext, SlidingWindowPolicy> windowPolicyProvider() {
    QueryParameterMetadata parameterMetadata = ((HazelcastRelOptCluster) getCluster()).getParameterMetadata();
    RexToExpressionVisitor visitor = new RexToExpressionVisitor(FAILING_FIELD_TYPE_PROVIDER, parameterMetadata);
    if (operator() == HazelcastSqlOperatorTable.TUMBLE) {
        Expression<?> windowSizeExpression = operand(2).accept(visitor);
        return context -> tumblingWinPolicy(WindowUtils.extractMillis(windowSizeExpression, context));
    } else if (operator() == HazelcastSqlOperatorTable.HOP) {
        Expression<?> windowSizeExpression = operand(2).accept(visitor);
        Expression<?> slideSizeExpression = operand(3).accept(visitor);
        return context -> slidingWinPolicy(WindowUtils.extractMillis(windowSizeExpression, context), WindowUtils.extractMillis(slideSizeExpression, context));
    } else {
        throw new IllegalArgumentException();
    }
}
Also used : SlidingWindowPolicy.slidingWinPolicy(com.hazelcast.jet.core.SlidingWindowPolicy.slidingWinPolicy) QueryParameterMetadata(com.hazelcast.sql.impl.QueryParameterMetadata) FAILING_FIELD_TYPE_PROVIDER(com.hazelcast.sql.impl.plan.node.PlanNodeFieldTypeProvider.FAILING_FIELD_TYPE_PROVIDER) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) HazelcastRelOptCluster(org.apache.calcite.plan.HazelcastRelOptCluster) RelColumnMapping(org.apache.calcite.rel.metadata.RelColumnMapping) RexNode(org.apache.calcite.rex.RexNode) SqlOperator(org.apache.calcite.sql.SqlOperator) RexToExpressionVisitor(com.hazelcast.jet.sql.impl.opt.physical.visitor.RexToExpressionVisitor) Expression(com.hazelcast.sql.impl.expression.Expression) Preconditions.checkTrue(com.hazelcast.internal.util.Preconditions.checkTrue) RelTraitSet(org.apache.calcite.plan.RelTraitSet) RelOptCluster(org.apache.calcite.plan.RelOptCluster) FunctionEx(com.hazelcast.function.FunctionEx) TableFunctionScan(org.apache.calcite.rel.core.TableFunctionScan) RelDataType(org.apache.calcite.rel.type.RelDataType) Set(java.util.Set) RelNode(org.apache.calcite.rel.RelNode) RexInputRef(org.apache.calcite.rex.RexInputRef) List(java.util.List) ExpressionEvalContext(com.hazelcast.sql.impl.expression.ExpressionEvalContext) Type(java.lang.reflect.Type) HazelcastSqlOperatorTable(com.hazelcast.jet.sql.impl.validate.HazelcastSqlOperatorTable) SlidingWindowPolicy.tumblingWinPolicy(com.hazelcast.jet.core.SlidingWindowPolicy.tumblingWinPolicy) WindowUtils(com.hazelcast.jet.sql.impl.aggregate.WindowUtils) RexCall(org.apache.calcite.rex.RexCall) HazelcastRelOptCluster(org.apache.calcite.plan.HazelcastRelOptCluster) RexToExpressionVisitor(com.hazelcast.jet.sql.impl.opt.physical.visitor.RexToExpressionVisitor) Expression(com.hazelcast.sql.impl.expression.Expression) QueryParameterMetadata(com.hazelcast.sql.impl.QueryParameterMetadata)

Aggregations

SlidingWindowPolicy (com.hazelcast.jet.core.SlidingWindowPolicy)14 Vertex (com.hazelcast.jet.core.Vertex)7 LongAccumulator (com.hazelcast.jet.accumulator.LongAccumulator)6 FunctionEx (com.hazelcast.function.FunctionEx)5 Entry (java.util.Map.Entry)5 ToLongFunctionEx (com.hazelcast.function.ToLongFunctionEx)4 AggregateOperation (com.hazelcast.jet.aggregate.AggregateOperation)4 Processor (com.hazelcast.jet.core.Processor)4 TimestampKind (com.hazelcast.jet.core.TimestampKind)4 Processors.combineToSlidingWindowP (com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP)4 List (java.util.List)4 Before (org.junit.Before)4 DAG (com.hazelcast.jet.core.DAG)3 SlidingWindowPolicy.slidingWinPolicy (com.hazelcast.jet.core.SlidingWindowPolicy.slidingWinPolicy)3 TimestampedEntry (com.hazelcast.jet.datamodel.TimestampedEntry)3 PlannerVertex (com.hazelcast.jet.impl.pipeline.Planner.PlannerVertex)3 Collections.singletonList (java.util.Collections.singletonList)3 Util.entry (com.hazelcast.jet.Util.entry)2 AggregateOperation1 (com.hazelcast.jet.aggregate.AggregateOperation1)2 Edge (com.hazelcast.jet.core.Edge)2