Search in sources :

Example 31 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class MetricsTest method customUnit_notUsed.

@Test
public void customUnit_notUsed() {
    pipeline.readFrom(TestSources.items(0L, 1L, 2L, 3L, 4L)).mapStateful(LongAccumulator::new, (acc, i) -> {
        acc.add(i);
        Metrics.metric("sum", Unit.COUNT);
        return acc.get();
    }).writeTo(Sinks.noop());
    Job job = runPipeline(pipeline.toDag());
    JobMetricsChecker checker = new JobMetricsChecker(job);
    checker.assertSummedMetricValue("sum", 0L);
}
Also used : ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) AggregateOperations.counting(com.hazelcast.jet.aggregate.AggregateOperations.counting) Arrays(java.util.Arrays) QuickTest(com.hazelcast.test.annotation.QuickTest) RunWith(org.junit.runner.RunWith) Processor(com.hazelcast.jet.core.Processor) Processors(com.hazelcast.jet.core.processor.Processors) CompletableFuture(java.util.concurrent.CompletableFuture) ServiceFactories.nonSharedService(com.hazelcast.jet.pipeline.ServiceFactories.nonSharedService) HashSet(java.util.HashSet) TestProcessors(com.hazelcast.jet.core.TestProcessors) Arrays.asList(java.util.Arrays.asList) DAG(com.hazelcast.jet.core.DAG) Edge(com.hazelcast.jet.core.Edge) Job(com.hazelcast.jet.Job) Before(org.junit.Before) JobRepository(com.hazelcast.jet.impl.JobRepository) Config(com.hazelcast.config.Config) HazelcastInstance(com.hazelcast.core.HazelcastInstance) FunctionEx(com.hazelcast.function.FunctionEx) Pipeline(com.hazelcast.jet.pipeline.Pipeline) JetTestSupport(com.hazelcast.jet.core.JetTestSupport) EXACTLY_ONCE(com.hazelcast.jet.config.ProcessingGuarantee.EXACTLY_ONCE) JobConfig(com.hazelcast.jet.config.JobConfig) Sinks(com.hazelcast.jet.pipeline.Sinks) Assert.assertTrue(org.junit.Assert.assertTrue) ResettableSingletonTraverser(com.hazelcast.jet.core.ResettableSingletonTraverser) Test(org.junit.Test) Category(org.junit.experimental.categories.Category) SupplierEx(com.hazelcast.function.SupplierEx) NoOutputSourceP(com.hazelcast.jet.core.TestProcessors.NoOutputSourceP) Assert.assertNotEquals(org.junit.Assert.assertNotEquals) Vertex(com.hazelcast.jet.core.Vertex) TestSources(com.hazelcast.jet.pipeline.test.TestSources) List(java.util.List) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Assert.assertFalse(org.junit.Assert.assertFalse) TransformP(com.hazelcast.jet.impl.processor.TransformP) FunctionEx.identity(com.hazelcast.function.FunctionEx.identity) HazelcastParallelClassRunner(com.hazelcast.test.HazelcastParallelClassRunner) RUNNING(com.hazelcast.jet.core.JobStatus.RUNNING) Assert.assertEquals(org.junit.Assert.assertEquals) Edge.between(com.hazelcast.jet.core.Edge.between) SinkProcessors.writeListP(com.hazelcast.jet.core.processor.SinkProcessors.writeListP) Job(com.hazelcast.jet.Job) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Example 32 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class OrderedStreamParallelismTest method applyTransformAndGetDag.

private DAG applyTransformAndGetDag(FunctionEx<StreamStage<Integer>, StreamStage<Integer>> transform) {
    PipelineImpl p = (PipelineImpl) Pipeline.create().setPreserveOrder(true);
    StreamStage<Integer> source = p.readFrom(TestSources.items(1)).setLocalParallelism(UPSTREAM_PARALLELISM).addTimestamps(t -> 0, Long.MAX_VALUE);
    StreamStage<Integer> applied = source.apply(transform);
    applied.mapStateful(LongAccumulator::new, (s, x) -> x).writeTo(Sinks.noop());
    return p.toDag(PIPELINE_CTX);
}
Also used : PipelineImpl(com.hazelcast.jet.impl.pipeline.PipelineImpl) FunctionEx(com.hazelcast.function.FunctionEx) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) Arrays(java.util.Arrays) HazelcastParametrizedRunner(com.hazelcast.test.HazelcastParametrizedRunner) QuickTest(com.hazelcast.test.annotation.QuickTest) Parameter(org.junit.runners.Parameterized.Parameter) Collection(java.util.Collection) RunWith(org.junit.runner.RunWith) Parameters(org.junit.runners.Parameterized.Parameters) Test(org.junit.Test) HazelcastSerialParametersRunnerFactory(com.hazelcast.test.HazelcastSerialParametersRunnerFactory) Traversers(com.hazelcast.jet.Traversers) Category(org.junit.experimental.categories.Category) Vertex(com.hazelcast.jet.core.Vertex) TestSources(com.hazelcast.jet.pipeline.test.TestSources) List(java.util.List) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) DAG(com.hazelcast.jet.core.DAG) Context(com.hazelcast.jet.impl.pipeline.PipelineImpl.Context) Collections(java.util.Collections) Assert.assertEquals(org.junit.Assert.assertEquals) UseParametersRunnerFactory(org.junit.runners.Parameterized.UseParametersRunnerFactory) PipelineImpl(com.hazelcast.jet.impl.pipeline.PipelineImpl)

Example 33 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class OrderedProcessingMultipleMemberTest method orderValidator.

private boolean orderValidator(LongAccumulator[] s, Long key, Map.Entry<Long, Long> entry) {
    LongAccumulator acc = s[key.intValue()];
    long value = entry.getValue();
    if (acc.get() >= value) {
        return false;
    } else {
        acc.set(value);
        return true;
    }
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator)

Example 34 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class AggregateOperation2Test method when_withIdentityFinish.

@Test
public void when_withIdentityFinish() {
    // Given
    AggregateOperation2<Object, Object, LongAccumulator, Long> aggrOp = AggregateOperation.withCreate(LongAccumulator::new).andAccumulate0((acc, item) -> acc.addAllowingOverflow(1)).andAccumulate1((acc, item) -> acc.addAllowingOverflow(1)).andExportFinish(LongAccumulator::get);
    // When
    AggregateOperation2<Object, Object, LongAccumulator, LongAccumulator> newAggrOp = aggrOp.withIdentityFinish();
    // Then
    LongAccumulator acc = newAggrOp.createFn().get();
    assertSame(acc, newAggrOp.finishFn().apply(acc));
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) FunctionEx(com.hazelcast.function.FunctionEx) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) AggregateOperations.aggregateOperation2(com.hazelcast.jet.aggregate.AggregateOperations.aggregateOperation2) QuickTest(com.hazelcast.test.annotation.QuickTest) RunWith(org.junit.runner.RunWith) BiConsumerEx(com.hazelcast.function.BiConsumerEx) Tag.tag2(com.hazelcast.jet.datamodel.Tag.tag2) Test(org.junit.Test) Category(org.junit.experimental.categories.Category) SupplierEx(com.hazelcast.function.SupplierEx) Tag.tag1(com.hazelcast.jet.datamodel.Tag.tag1) Tag.tag0(com.hazelcast.jet.datamodel.Tag.tag0) Assert.assertSame(org.junit.Assert.assertSame) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) HazelcastParallelClassRunner(com.hazelcast.test.HazelcastParallelClassRunner) AggregateOperations.summingLong(com.hazelcast.jet.aggregate.AggregateOperations.summingLong) Functions.wholeItem(com.hazelcast.function.Functions.wholeItem) Tuple2(com.hazelcast.jet.datamodel.Tuple2) Assert.assertEquals(org.junit.Assert.assertEquals) AggregateOperations.summingLong(com.hazelcast.jet.aggregate.AggregateOperations.summingLong) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Example 35 with LongAccumulator

use of com.hazelcast.jet.accumulator.LongAccumulator in project hazelcast by hazelcast.

the class AggregateOperation1Test method when_andThen_then_exportAndFinishChanged.

@Test
public void when_andThen_then_exportAndFinishChanged() {
    // Given
    AggregateOperation1<Long, LongAccumulator, Long> aggrOp = summingLong((Long x) -> x);
    // When
    AggregateOperation1<Long, LongAccumulator, Long> incAggrOp = aggrOp.andThen(a -> a + 1);
    // Then
    LongAccumulator acc = incAggrOp.createFn().get();
    incAggrOp.accumulateFn().accept(acc, 13L);
    long exported = incAggrOp.exportFn().apply(acc);
    long finished = incAggrOp.finishFn().apply(acc);
    assertEquals(14L, exported);
    assertEquals(14L, finished);
}
Also used : LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) AggregateOperations.summingLong(com.hazelcast.jet.aggregate.AggregateOperations.summingLong) ParallelJVMTest(com.hazelcast.test.annotation.ParallelJVMTest) QuickTest(com.hazelcast.test.annotation.QuickTest) Test(org.junit.Test)

Aggregations

LongAccumulator (com.hazelcast.jet.accumulator.LongAccumulator)43 Test (org.junit.Test)32 Category (org.junit.experimental.categories.Category)24 RunWith (org.junit.runner.RunWith)20 QuickTest (com.hazelcast.test.annotation.QuickTest)19 Arrays (java.util.Arrays)17 List (java.util.List)17 Assert.assertEquals (org.junit.Assert.assertEquals)17 Job (com.hazelcast.jet.Job)16 ParallelJVMTest (com.hazelcast.test.annotation.ParallelJVMTest)16 HazelcastInstance (com.hazelcast.core.HazelcastInstance)14 FunctionEx (com.hazelcast.function.FunctionEx)14 Pipeline (com.hazelcast.jet.pipeline.Pipeline)14 Util.entry (com.hazelcast.jet.Util.entry)13 HazelcastParallelClassRunner (com.hazelcast.test.HazelcastParallelClassRunner)13 Entry (java.util.Map.Entry)13 SupplierEx (com.hazelcast.function.SupplierEx)11 Sinks (com.hazelcast.jet.pipeline.Sinks)11 Tag.tag0 (com.hazelcast.jet.datamodel.Tag.tag0)10 Tag.tag1 (com.hazelcast.jet.datamodel.Tag.tag1)10