Search in sources :

Example 6 with IMapJet

use of com.hazelcast.jet.IMapJet in project hazelcast-jet-reference-manual by hazelcast.

the class JUS method s1.

static void s1() {
    JetInstance jet = Jet.newJetInstance();
    try {
        // tag::s1[]
        IMapJet<String, Integer> map = jet.getMap("latitudes");
        map.put("London", 51);
        map.put("Paris", 48);
        map.put("NYC", 40);
        map.put("Sydney", -34);
        map.put("Sao Paulo", -23);
        map.put("Jakarta", -6);
        DistributedStream.fromMap(map).filter(e -> e.getValue() < 0).forEach(System.out::println);
    // end::s1[]
    } finally {
        Jet.shutdownAll();
    }
}
Also used : Arrays(java.util.Arrays) JetInstance(com.hazelcast.jet.JetInstance) IMapJet(com.hazelcast.jet.IMapJet) Collectors(java.util.stream.Collectors) DistributedFunctions.wholeItem(com.hazelcast.jet.function.DistributedFunctions.wholeItem) Sources(com.hazelcast.jet.pipeline.Sources) IMap(com.hazelcast.core.IMap) DistributedCollectors(com.hazelcast.jet.stream.DistributedCollectors) Stream(java.util.stream.Stream) Map(java.util.Map) IList(com.hazelcast.core.IList) DistributedStream(com.hazelcast.jet.stream.DistributedStream) Jet(com.hazelcast.jet.Jet) DistributedCollectors.toMap(com.hazelcast.jet.stream.DistributedCollectors.toMap) JetInstance(com.hazelcast.jet.JetInstance)

Example 7 with IMapJet

use of com.hazelcast.jet.IMapJet in project hazelcast-jet by hazelcast.

the class HazelcastConnectorTest method when_streamMap_withFilterAndProjection.

@Test
public void when_streamMap_withFilterAndProjection() {
    DAG dag = new DAG();
    Vertex source = dag.newVertex("source", SourceProcessors.<Integer, Integer, Integer>streamMapP(streamSourceName, event -> event.getKey() != 0, EventJournalMapEvent::getKey, START_FROM_OLDEST, wmGenParams(i -> i, limitingLag(0), noThrottling(), 10_000)));
    Vertex sink = dag.newVertex("sink", writeListP(streamSinkName));
    dag.edge(between(source, sink));
    Job job = jetInstance.newJob(dag);
    IMapJet<Integer, Integer> sourceMap = jetInstance.getMap(streamSourceName);
    range(0, ENTRY_COUNT).forEach(i -> sourceMap.put(i, i));
    assertSizeEventually(ENTRY_COUNT - 1, jetInstance.getList(streamSinkName));
    assertFalse(jetInstance.getList(streamSinkName).contains(0));
    assertTrue(jetInstance.getList(streamSinkName).contains(1));
    job.cancel();
}
Also used : WatermarkPolicies.limitingLag(com.hazelcast.jet.core.WatermarkPolicies.limitingLag) IntStream.range(java.util.stream.IntStream.range) EventJournalConfig(com.hazelcast.config.EventJournalConfig) IMapJet(com.hazelcast.jet.IMapJet) SourceProcessors.readMapP(com.hazelcast.jet.core.processor.SourceProcessors.readMapP) CacheSimpleConfig(com.hazelcast.config.CacheSimpleConfig) SourceProcessors.readListP(com.hazelcast.jet.core.processor.SourceProcessors.readListP) SourceProcessors.streamCacheP(com.hazelcast.jet.core.processor.SourceProcessors.streamCacheP) ICacheJet(com.hazelcast.jet.ICacheJet) Map(java.util.Map) WatermarkEmissionPolicy.noThrottling(com.hazelcast.jet.core.WatermarkEmissionPolicy.noThrottling) SinkProcessors.writeCacheP(com.hazelcast.jet.core.processor.SinkProcessors.writeCacheP) DAG(com.hazelcast.jet.core.DAG) WatermarkGenerationParams.noWatermarks(com.hazelcast.jet.core.WatermarkGenerationParams.noWatermarks) Projections(com.hazelcast.projection.Projections) WatermarkGenerationParams.wmGenParams(com.hazelcast.jet.core.WatermarkGenerationParams.wmGenParams) SourceProcessors.readCacheP(com.hazelcast.jet.core.processor.SourceProcessors.readCacheP) EventJournalMapEvent(com.hazelcast.map.journal.EventJournalMapEvent) JetTestSupport(com.hazelcast.jet.core.JetTestSupport) START_FROM_OLDEST(com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_OLDEST) Collectors.joining(java.util.stream.Collectors.joining) PredicateTestUtils.entry(com.hazelcast.query.impl.predicates.PredicateTestUtils.entry) JetCacheManager(com.hazelcast.jet.JetCacheManager) Assert.assertFalse(org.junit.Assert.assertFalse) HazelcastParallelClassRunner(com.hazelcast.test.HazelcastParallelClassRunner) Entry(java.util.Map.Entry) SinkProcessors.writeMapP(com.hazelcast.jet.core.processor.SinkProcessors.writeMapP) IntStream(java.util.stream.IntStream) JetInstance(com.hazelcast.jet.JetInstance) SourceProcessors(com.hazelcast.jet.core.processor.SourceProcessors) RunWith(org.junit.runner.RunWith) Job(com.hazelcast.jet.Job) Before(org.junit.Before) EventJournalCacheEvent(com.hazelcast.cache.journal.EventJournalCacheEvent) Config(com.hazelcast.config.Config) JetConfig(com.hazelcast.jet.config.JetConfig) SourceProcessors.streamMapP(com.hazelcast.jet.core.processor.SourceProcessors.streamMapP) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) IListJet(com.hazelcast.jet.IListJet) Vertex(com.hazelcast.jet.core.Vertex) TruePredicate(com.hazelcast.query.TruePredicate) Collectors.toList(java.util.stream.Collectors.toList) Predicates(com.hazelcast.query.Predicates) ICache(com.hazelcast.cache.ICache) Util.mapPutEvents(com.hazelcast.jet.Util.mapPutEvents) Assert.assertEquals(org.junit.Assert.assertEquals) Edge.between(com.hazelcast.jet.core.Edge.between) SinkProcessors.writeListP(com.hazelcast.jet.core.processor.SinkProcessors.writeListP) Vertex(com.hazelcast.jet.core.Vertex) DAG(com.hazelcast.jet.core.DAG) Job(com.hazelcast.jet.Job) Test(org.junit.Test)

Example 8 with IMapJet

use of com.hazelcast.jet.IMapJet in project hazelcast-jet by hazelcast.

the class SnapshotFailureTest method when_snapshotFails_then_jobShouldNotFail.

@Test
public void when_snapshotFails_then_jobShouldNotFail() {
    int numPartitions = 2;
    int numElements = 10;
    IMapJet<Object, Object> results = instance1.getMap("results");
    DAG dag = new DAG();
    SequencesInPartitionsMetaSupplier sup = new SequencesInPartitionsMetaSupplier(numPartitions, numElements);
    Vertex generator = dag.newVertex("generator", peekOutputP(throttle(sup, 2))).localParallelism(1);
    Vertex writeMap = dag.newVertex("writeMap", writeMapP(results.getName())).localParallelism(1);
    dag.edge(between(generator, writeMap));
    JobConfig config = new JobConfig();
    config.setProcessingGuarantee(ProcessingGuarantee.EXACTLY_ONCE);
    config.setSnapshotIntervalMillis(100);
    Job job = instance1.newJob(dag, config);
    // let's start a second job that will watch the snapshots map and write failed
    // SnapshotRecords to a list, which we will check for presence of failed snapshot
    Pipeline p = Pipeline.create();
    p.drawFrom(Sources.mapJournal(snapshotsMapName(job.getId()), event -> event.getNewValue() instanceof SnapshotRecord && ((SnapshotRecord) event.getNewValue()).status() == SnapshotStatus.FAILED, EventJournalMapEvent::getNewValue, JournalInitialPosition.START_FROM_OLDEST)).peek().drainTo(Sinks.list("failed_snapshot_records"));
    instance1.newJob(p);
    job.join();
    assertEquals("numPartitions", numPartitions, results.size());
    assertEquals("offset partition 0", numElements - 1, results.get(0));
    assertEquals("offset partition 1", numElements - 1, results.get(1));
    assertTrue("no failure occurred in store", storeFailed);
    assertFalse("no failed snapshot appeared in snapshotsMap", instance1.getList("failed_snapshot_records").isEmpty());
}
Also used : JetInstance(com.hazelcast.jet.JetInstance) SnapshotRecord(com.hazelcast.jet.impl.execution.SnapshotRecord) RunWith(org.junit.runner.RunWith) EventJournalConfig(com.hazelcast.config.EventJournalConfig) IMapJet(com.hazelcast.jet.IMapJet) SequencesInPartitionsMetaSupplier(com.hazelcast.jet.core.JobRestartWithSnapshotTest.SequencesInPartitionsMetaSupplier) SnapshotStatus(com.hazelcast.jet.impl.execution.SnapshotRecord.SnapshotStatus) HazelcastSerialClassRunner(com.hazelcast.test.HazelcastSerialClassRunner) MapConfig(com.hazelcast.config.MapConfig) TestUtil.throttle(com.hazelcast.jet.core.TestUtil.throttle) ExpectedException(org.junit.rules.ExpectedException) Job(com.hazelcast.jet.Job) EventJournalMapEvent(com.hazelcast.map.journal.EventJournalMapEvent) Before(org.junit.Before) JetConfig(com.hazelcast.jet.config.JetConfig) SnapshotRepository(com.hazelcast.jet.impl.SnapshotRepository) Pipeline(com.hazelcast.jet.pipeline.Pipeline) JobConfig(com.hazelcast.jet.config.JobConfig) Sinks(com.hazelcast.jet.pipeline.Sinks) Assert.assertTrue(org.junit.Assert.assertTrue) Test(org.junit.Test) DiagnosticProcessors.peekOutputP(com.hazelcast.jet.core.processor.DiagnosticProcessors.peekOutputP) AMapStore(com.hazelcast.client.map.helpers.AMapStore) Serializable(java.io.Serializable) Sources(com.hazelcast.jet.pipeline.Sources) Rule(org.junit.Rule) JournalInitialPosition(com.hazelcast.jet.pipeline.JournalInitialPosition) Assert.assertFalse(org.junit.Assert.assertFalse) MapStoreConfig(com.hazelcast.config.MapStoreConfig) ProcessingGuarantee(com.hazelcast.jet.config.ProcessingGuarantee) SnapshotRepository.snapshotsMapName(com.hazelcast.jet.impl.SnapshotRepository.snapshotsMapName) Assert.assertEquals(org.junit.Assert.assertEquals) Edge.between(com.hazelcast.jet.core.Edge.between) SinkProcessors.writeMapP(com.hazelcast.jet.core.processor.SinkProcessors.writeMapP) SnapshotRecord(com.hazelcast.jet.impl.execution.SnapshotRecord) Job(com.hazelcast.jet.Job) SequencesInPartitionsMetaSupplier(com.hazelcast.jet.core.JobRestartWithSnapshotTest.SequencesInPartitionsMetaSupplier) JobConfig(com.hazelcast.jet.config.JobConfig) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Test(org.junit.Test)

Aggregations

IMapJet (com.hazelcast.jet.IMapJet)8 JetInstance (com.hazelcast.jet.JetInstance)8 IMap (com.hazelcast.core.IMap)5 Job (com.hazelcast.jet.Job)5 JetConfig (com.hazelcast.jet.config.JetConfig)5 Edge.between (com.hazelcast.jet.core.Edge.between)5 Map (java.util.Map)5 Assert.assertEquals (org.junit.Assert.assertEquals)5 Assert.assertFalse (org.junit.Assert.assertFalse)5 Assert.assertTrue (org.junit.Assert.assertTrue)5 Before (org.junit.Before)5 Test (org.junit.Test)5 RunWith (org.junit.runner.RunWith)5 WatermarkGenerationParams.wmGenParams (com.hazelcast.jet.core.WatermarkGenerationParams.wmGenParams)4 WatermarkPolicies.limitingLag (com.hazelcast.jet.core.WatermarkPolicies.limitingLag)4 SinkProcessors.writeListP (com.hazelcast.jet.core.processor.SinkProcessors.writeListP)4 Arrays (java.util.Arrays)4 Entry (java.util.Map.Entry)4 JobConfig (com.hazelcast.jet.config.JobConfig)3 ProcessingGuarantee (com.hazelcast.jet.config.ProcessingGuarantee)3