Search in sources :

Example 16 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class OrderedProcessingMergingStagesTest method orderValidator.

private boolean orderValidator(LongAccumulator[] s, Long key, Map.Entry<Long, Long> entry) {
    LongAccumulator acc = s[key.intValue()];
    long value = entry.getValue();
    if (acc.get() >= value) {
        return false;
    } else {
        acc.set(value);
        return true;
    }
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator)

Example 17 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class SlidingWindowPTest method before.

@Before
public void before() {
    SlidingWindowPolicy winPolicy = slidingWinPolicy(4, 1);
    AggregateOperation1<Entry<?, Long>, LongAccumulator, Long> operation = AggregateOperation.withCreate(LongAccumulator::new).andAccumulate((LongAccumulator acc, Entry<?, Long> item) -> acc.add(item.getValue())).andCombine(LongAccumulator::add).andDeduct(hasDeduct ? LongAccumulator::subtract : null).andExportFinish(LongAccumulator::get);
    FunctionEx<?, Long> keyFn = t -> KEY;
    ToLongFunctionEx<Entry<Long, Long>> timestampFn = Entry::getKey;
    SupplierEx<Processor> procSupplier = singleStageProcessor ? aggregateToSlidingWindowP(singletonList(keyFn), singletonList(timestampFn), TimestampKind.EVENT, winPolicy, 0L, operation, KeyedWindowResult::new) : combineToSlidingWindowP(winPolicy, operation, KeyedWindowResult::new);
    // new supplier to save the last supplied instance
    supplier = () -> lastSuppliedProcessor = (SlidingWindowP) procSupplier.get();
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Collections.shuffle(java.util.Collections.shuffle) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) Arrays(java.util.Arrays) KeyedWindowResult(com.hazelcast.jet.datamodel.KeyedWindowResult) QuickTest(com.hazelcast.test.annotation.QuickTest) RunWith(org.junit.runner.RunWith) Parameters(org.junit.runners.Parameterized.Parameters) Processor(com.hazelcast.jet.core.Processor) TimestampKind(com.hazelcast.jet.core.TimestampKind) TestSupport.verifyProcessor(com.hazelcast.jet.core.test.TestSupport.verifyProcessor) SlidingWindowPolicy.slidingWinPolicy(com.hazelcast.jet.core.SlidingWindowPolicy.slidingWinPolicy) ArrayList(java.util.ArrayList) Collections.singletonList(java.util.Collections.singletonList) Watermark(com.hazelcast.jet.core.Watermark) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Processors.aggregateToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.aggregateToSlidingWindowP) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) Util.entry(com.hazelcast.jet.Util.entry) Arrays.asList(java.util.Arrays.asList) After(org.junit.After) JetTestSupport.wm(com.hazelcast.jet.core.JetTestSupport.wm) Processors.combineToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP) ExpectedException(org.junit.rules.ExpectedException) Before(org.junit.Before) UseParametersRunnerFactory(org.junit.runners.Parameterized.UseParametersRunnerFactory) FunctionEx(com.hazelcast.function.FunctionEx) LongStream(java.util.stream.LongStream) HazelcastParametrizedRunner(com.hazelcast.test.HazelcastParametrizedRunner) Parameter(org.junit.runners.Parameterized.Parameter) Collection(java.util.Collection) Assert.assertTrue(org.junit.Assert.assertTrue) HazelcastParallelParametersRunnerFactory(com.hazelcast.test.HazelcastParallelParametersRunnerFactory) Test(org.junit.Test) AggregateOperation1(com.hazelcast.jet.aggregate.AggregateOperation1) Category(org.junit.experimental.categories.Category) SupplierEx(com.hazelcast.function.SupplierEx) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) Rule(org.junit.Rule) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Entry(java.util.Map.Entry) Entry(java.util.Map.Entry) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Processor(com.hazelcast.jet.core.Processor) TestSupport.verifyProcessor(com.hazelcast.jet.core.test.TestSupport.verifyProcessor) Processors.aggregateToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.aggregateToSlidingWindowP) Processors.combineToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP) Before(org.junit.Before)

Example 18 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class SlidingWindowP_failoverTest method init.

private void init(ProcessingGuarantee guarantee) throws Exception {
    SlidingWindowPolicy wDef = SlidingWindowPolicy.tumblingWinPolicy(1);
    AggregateOperation1<Object, LongAccumulator, Long> aggrOp = counting();
    p = new SlidingWindowP<>(singletonList(entryKey()), singletonList((ToLongFunctionEx<Entry<?, Long>>) Entry::getValue), wDef, 0L, aggrOp, KeyedWindowResult::new, true);
    Outbox outbox = new TestOutbox(128);
    Context context = new TestProcessorContext().setProcessingGuarantee(guarantee);
    p.init(outbox, context);
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) TestProcessorContext(com.hazelcast.jet.core.test.TestProcessorContext) Context(com.hazelcast.jet.core.Processor.Context) Entry(java.util.Map.Entry) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) TestOutbox(com.hazelcast.jet.core.test.TestOutbox) Outbox(com.hazelcast.jet.core.Outbox) TestOutbox(com.hazelcast.jet.core.test.TestOutbox) TestProcessorContext(com.hazelcast.jet.core.test.TestProcessorContext)

Example 19 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class SlidingWindowP_twoStageSnapshotTest method before.

@Before
public void before() {
    SlidingWindowPolicy windowDef = slidingWinPolicy(4, 1);
    AggregateOperation1<Entry<?, Long>, LongAccumulator, Long> aggrOp = AggregateOperation.withCreate(LongAccumulator::new).andAccumulate((LongAccumulator acc, Entry<?, Long> item) -> acc.add(item.getValue())).andCombine(LongAccumulator::add).andDeduct(LongAccumulator::subtract).andExportFinish(LongAccumulator::get);
    SupplierEx<Processor> procSupplier1 = Processors.accumulateByFrameP(singletonList((FunctionEx<? super Entry<Long, Long>, ?>) t -> KEY), singletonList((ToLongFunctionEx<? super Entry<Long, Long>>) Entry::getKey), TimestampKind.EVENT, windowDef, aggrOp.withIdentityFinish());
    SupplierEx<Processor> procSupplier2 = combineToSlidingWindowP(windowDef, aggrOp, KeyedWindowResult::new);
    // new supplier to save the last supplied instance
    stage1Supplier = () -> lastSuppliedStage1Processor = (SlidingWindowP<?, ?, ?, ?>) procSupplier1.get();
    stage2Supplier = () -> lastSuppliedStage2Processor = (SlidingWindowP<?, ?, ?, ?>) procSupplier2.get();
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Entry(java.util.Map.Entry) SlidingWindowPolicy(com.hazelcast.jet.core.SlidingWindowPolicy) Processor(com.hazelcast.jet.core.Processor) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) FunctionEx(com.hazelcast.function.FunctionEx) ToLongFunctionEx(com.hazelcast.function.ToLongFunctionEx) Processors.combineToSlidingWindowP(com.hazelcast.jet.core.processor.Processors.combineToSlidingWindowP) KeyedWindowResult(com.hazelcast.jet.datamodel.KeyedWindowResult) Before(org.junit.Before)

Example 20 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class AggregateOperation1Test method when_withIdentityFinish.

@Test
public void when_withIdentityFinish() {
    // Given
    AggregateOperation1<Object, LongAccumulator, Long> aggrOp = AggregateOperation.withCreate(LongAccumulator::new).andAccumulate((acc, item) -> acc.addAllowingOverflow(1)).andExportFinish(LongAccumulator::get);
    // When
    AggregateOperation<LongAccumulator, LongAccumulator> newAggrOp = aggrOp.withIdentityFinish();
    // Then
    LongAccumulator acc = newAggrOp.createFn().get();
    assertSame(acc, newAggrOp.finishFn().apply(acc));
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) FunctionEx(com.hazelcast.function.FunctionEx) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) RunWith(org.junit.runner.RunWith) BiConsumerEx(com.hazelcast.function.BiConsumerEx) Test(org.junit.Test) Category(org.junit.experimental.categories.Category) SupplierEx(com.hazelcast.function.SupplierEx) Tag.tag1(com.hazelcast.jet.datamodel.Tag.tag1) Tag.tag0(com.hazelcast.jet.datamodel.Tag.tag0) Assert.assertSame(org.junit.Assert.assertSame) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) HazelcastParallelClassRunner(com.hazelcast.test.HazelcastParallelClassRunner) AggregateOperations.summingLong(com.hazelcast.jet.aggregate.AggregateOperations.summingLong) Functions.wholeItem(com.hazelcast.function.Functions.wholeItem) Assert.assertEquals(org.junit.Assert.assertEquals) AggregateOperations.summingLong(com.hazelcast.jet.aggregate.AggregateOperations.summingLong) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Aggregations

LongAccumulator (com.hazelcast.jet.accumulator.LongAccumulator)43 Test (org.junit.Test)32 Category (org.junit.experimental.categories.Category)24 RunWith (org.junit.runner.RunWith)20 QuickTest (com.hazelcast.test.annotation.QuickTest)19 Arrays (java.util.Arrays)17 List (java.util.List)17 Assert.assertEquals (org.junit.Assert.assertEquals)17 Job (com.hazelcast.jet.Job)16 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)16 HazelcastInstance (com.hazelcast.core.HazelcastInstance)14 FunctionEx (com.hazelcast.function.FunctionEx)14 Pipeline (com.hazelcast.jet.pipeline.Pipeline)14 Util.entry (com.hazelcast.jet.Util.entry)13 HazelcastParallelClassRunner (com.hazelcast.test.HazelcastParallelClassRunner)13 Entry (java.util.Map.Entry)13 SupplierEx (com.hazelcast.function.SupplierEx)11 Sinks (com.hazelcast.jet.pipeline.Sinks)11 Tag.tag0 (com.hazelcast.jet.datamodel.Tag.tag0)10 Tag.tag1 (com.hazelcast.jet.datamodel.Tag.tag1)10