Search in sources :

Example 1 with START_FROM_CURRENT

use of com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT in project hazelcast-jet-reference-manual by hazelcast.

the class BuildComputation method s14.

static void s14() {
    // tag::s14[]
    Pipeline p = Pipeline.create();
    p.<Tweet>drawFrom(Sources.mapJournal("tweets", mapPutEvents(), mapEventNewValue(), START_FROM_CURRENT)).flatMap(tweet -> traverseArray(tweet.text().toLowerCase().split("\\W+")).map(word -> new TweetWord(tweet.timestamp(), word))).filter(tweetWord -> !tweetWord.word().isEmpty()).addTimestamps(TweetWord::timestamp, TimeUnit.SECONDS.toMillis(5)).window(sliding(MINUTES.toMillis(1), SECONDS.toMillis(1))).groupingKey(TweetWord::word).aggregate(counting()).drainTo(Sinks.list("result"));
// end::s14[]
}
Also used : AggregateOperations.counting(com.hazelcast.jet.aggregate.AggregateOperations.counting) Delivery(datamodel.Delivery) MINUTES(java.util.concurrent.TimeUnit.MINUTES) Tweet(datamodel.Tweet) JoinClause.joinMapEntries(com.hazelcast.jet.pipeline.JoinClause.joinMapEntries) DistributedFunctions.wholeItem(com.hazelcast.jet.function.DistributedFunctions.wholeItem) BatchStage(com.hazelcast.jet.pipeline.BatchStage) Util.mapEventNewValue(com.hazelcast.jet.Util.mapEventNewValue) PageVisit(datamodel.PageVisit) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) START_FROM_CURRENT(com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT) DAG(com.hazelcast.jet.core.DAG) StreamHashJoinBuilder(com.hazelcast.jet.pipeline.StreamHashJoinBuilder) Tuple2(com.hazelcast.jet.datamodel.Tuple2) AddToCart(datamodel.AddToCart) Tuple3(com.hazelcast.jet.datamodel.Tuple3) StreamStage(com.hazelcast.jet.pipeline.StreamStage) StageWithGrouping(com.hazelcast.jet.pipeline.StageWithGrouping) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Market(datamodel.Market) AggregateOperation2(com.hazelcast.jet.aggregate.AggregateOperation2) Tag(com.hazelcast.jet.datamodel.Tag) Sinks(com.hazelcast.jet.pipeline.Sinks) Payment(datamodel.Payment) Sources(com.hazelcast.jet.pipeline.Sources) Product(datamodel.Product) TimeUnit(java.util.concurrent.TimeUnit) BagsByTag(com.hazelcast.jet.datamodel.BagsByTag) WindowDefinition.sliding(com.hazelcast.jet.pipeline.WindowDefinition.sliding) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Traversers.traverseArray(com.hazelcast.jet.Traversers.traverseArray) ItemsByTag(com.hazelcast.jet.datamodel.ItemsByTag) Entry(java.util.Map.Entry) Trade(datamodel.Trade) TweetWord(datamodel.TweetWord) Util.mapPutEvents(com.hazelcast.jet.Util.mapPutEvents) AggregateOperations.toBagsByTag(com.hazelcast.jet.aggregate.AggregateOperations.toBagsByTag) GroupAggregateBuilder(com.hazelcast.jet.pipeline.GroupAggregateBuilder) SECONDS(java.util.concurrent.TimeUnit.SECONDS) Broker(datamodel.Broker) TweetWord(datamodel.TweetWord) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Example 2 with START_FROM_CURRENT

use of com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT in project hazelcast-jet-reference-manual by hazelcast.

the class BuildComputation method s15.

static void s15() {
    // tag::s15[]
    Pipeline p = Pipeline.create();
    p.<Tweet>drawFrom(Sources.mapJournal("tweets", mapPutEvents(), mapEventNewValue(), START_FROM_CURRENT)).addTimestamps(Tweet::timestamp, TimeUnit.SECONDS.toMillis(5)).flatMap(tweet -> traverseArray(tweet.text().toLowerCase().split("\\W+"))).filter(word -> !word.isEmpty()).window(sliding(MINUTES.toMillis(1), SECONDS.toMillis(1))).groupingKey(wholeItem()).aggregate(counting()).drainTo(Sinks.list("result"));
// end::s15[]
}
Also used : AggregateOperations.counting(com.hazelcast.jet.aggregate.AggregateOperations.counting) Delivery(datamodel.Delivery) MINUTES(java.util.concurrent.TimeUnit.MINUTES) Tweet(datamodel.Tweet) JoinClause.joinMapEntries(com.hazelcast.jet.pipeline.JoinClause.joinMapEntries) DistributedFunctions.wholeItem(com.hazelcast.jet.function.DistributedFunctions.wholeItem) BatchStage(com.hazelcast.jet.pipeline.BatchStage) Util.mapEventNewValue(com.hazelcast.jet.Util.mapEventNewValue) PageVisit(datamodel.PageVisit) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) START_FROM_CURRENT(com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT) DAG(com.hazelcast.jet.core.DAG) StreamHashJoinBuilder(com.hazelcast.jet.pipeline.StreamHashJoinBuilder) Tuple2(com.hazelcast.jet.datamodel.Tuple2) AddToCart(datamodel.AddToCart) Tuple3(com.hazelcast.jet.datamodel.Tuple3) StreamStage(com.hazelcast.jet.pipeline.StreamStage) StageWithGrouping(com.hazelcast.jet.pipeline.StageWithGrouping) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Market(datamodel.Market) AggregateOperation2(com.hazelcast.jet.aggregate.AggregateOperation2) Tag(com.hazelcast.jet.datamodel.Tag) Sinks(com.hazelcast.jet.pipeline.Sinks) Payment(datamodel.Payment) Sources(com.hazelcast.jet.pipeline.Sources) Product(datamodel.Product) TimeUnit(java.util.concurrent.TimeUnit) BagsByTag(com.hazelcast.jet.datamodel.BagsByTag) WindowDefinition.sliding(com.hazelcast.jet.pipeline.WindowDefinition.sliding) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Traversers.traverseArray(com.hazelcast.jet.Traversers.traverseArray) ItemsByTag(com.hazelcast.jet.datamodel.ItemsByTag) Entry(java.util.Map.Entry) Trade(datamodel.Trade) TweetWord(datamodel.TweetWord) Util.mapPutEvents(com.hazelcast.jet.Util.mapPutEvents) AggregateOperations.toBagsByTag(com.hazelcast.jet.aggregate.AggregateOperations.toBagsByTag) GroupAggregateBuilder(com.hazelcast.jet.pipeline.GroupAggregateBuilder) SECONDS(java.util.concurrent.TimeUnit.SECONDS) Broker(datamodel.Broker) Tweet(datamodel.Tweet) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Example 3 with START_FROM_CURRENT

use of com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT in project hazelcast-jet-reference-manual by hazelcast.

the class BuildComputation method s13.

static void s13() {
    // tag::s13[]
    Pipeline p = Pipeline.create();
    p.<String>drawFrom(Sources.mapJournal("tweets", mapPutEvents(), mapEventNewValue(), START_FROM_CURRENT)).flatMap(line -> traverseArray(line.toLowerCase().split("\\W+"))).filter(word -> !word.isEmpty()).addTimestamps().window(sliding(MINUTES.toMillis(1), SECONDS.toMillis(1))).groupingKey(wholeItem()).aggregate(counting()).drainTo(Sinks.list("result"));
// end::s13[]
}
Also used : AggregateOperations.counting(com.hazelcast.jet.aggregate.AggregateOperations.counting) Delivery(datamodel.Delivery) MINUTES(java.util.concurrent.TimeUnit.MINUTES) Tweet(datamodel.Tweet) JoinClause.joinMapEntries(com.hazelcast.jet.pipeline.JoinClause.joinMapEntries) DistributedFunctions.wholeItem(com.hazelcast.jet.function.DistributedFunctions.wholeItem) BatchStage(com.hazelcast.jet.pipeline.BatchStage) Util.mapEventNewValue(com.hazelcast.jet.Util.mapEventNewValue) PageVisit(datamodel.PageVisit) AggregateOperation(com.hazelcast.jet.aggregate.AggregateOperation) START_FROM_CURRENT(com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT) DAG(com.hazelcast.jet.core.DAG) StreamHashJoinBuilder(com.hazelcast.jet.pipeline.StreamHashJoinBuilder) Tuple2(com.hazelcast.jet.datamodel.Tuple2) AddToCart(datamodel.AddToCart) Tuple3(com.hazelcast.jet.datamodel.Tuple3) StreamStage(com.hazelcast.jet.pipeline.StreamStage) StageWithGrouping(com.hazelcast.jet.pipeline.StageWithGrouping) Pipeline(com.hazelcast.jet.pipeline.Pipeline) Market(datamodel.Market) AggregateOperation2(com.hazelcast.jet.aggregate.AggregateOperation2) Tag(com.hazelcast.jet.datamodel.Tag) Sinks(com.hazelcast.jet.pipeline.Sinks) Payment(datamodel.Payment) Sources(com.hazelcast.jet.pipeline.Sources) Product(datamodel.Product) TimeUnit(java.util.concurrent.TimeUnit) BagsByTag(com.hazelcast.jet.datamodel.BagsByTag) WindowDefinition.sliding(com.hazelcast.jet.pipeline.WindowDefinition.sliding) LongAccumulator(com.hazelcast.jet.accumulator.LongAccumulator) Traversers.traverseArray(com.hazelcast.jet.Traversers.traverseArray) ItemsByTag(com.hazelcast.jet.datamodel.ItemsByTag) Entry(java.util.Map.Entry) Trade(datamodel.Trade) TweetWord(datamodel.TweetWord) Util.mapPutEvents(com.hazelcast.jet.Util.mapPutEvents) AggregateOperations.toBagsByTag(com.hazelcast.jet.aggregate.AggregateOperations.toBagsByTag) GroupAggregateBuilder(com.hazelcast.jet.pipeline.GroupAggregateBuilder) SECONDS(java.util.concurrent.TimeUnit.SECONDS) Broker(datamodel.Broker) Pipeline(com.hazelcast.jet.pipeline.Pipeline)

Aggregations

Traversers.traverseArray (com.hazelcast.jet.Traversers.traverseArray)3 Util.mapEventNewValue (com.hazelcast.jet.Util.mapEventNewValue)3 Util.mapPutEvents (com.hazelcast.jet.Util.mapPutEvents)3 LongAccumulator (com.hazelcast.jet.accumulator.LongAccumulator)3 AggregateOperation (com.hazelcast.jet.aggregate.AggregateOperation)3 AggregateOperation2 (com.hazelcast.jet.aggregate.AggregateOperation2)3 AggregateOperations.counting (com.hazelcast.jet.aggregate.AggregateOperations.counting)3 AggregateOperations.toBagsByTag (com.hazelcast.jet.aggregate.AggregateOperations.toBagsByTag)3 DAG (com.hazelcast.jet.core.DAG)3 BagsByTag (com.hazelcast.jet.datamodel.BagsByTag)3 ItemsByTag (com.hazelcast.jet.datamodel.ItemsByTag)3 Tag (com.hazelcast.jet.datamodel.Tag)3 Tuple2 (com.hazelcast.jet.datamodel.Tuple2)3 Tuple3 (com.hazelcast.jet.datamodel.Tuple3)3 DistributedFunctions.wholeItem (com.hazelcast.jet.function.DistributedFunctions.wholeItem)3 BatchStage (com.hazelcast.jet.pipeline.BatchStage)3 GroupAggregateBuilder (com.hazelcast.jet.pipeline.GroupAggregateBuilder)3 JoinClause.joinMapEntries (com.hazelcast.jet.pipeline.JoinClause.joinMapEntries)3 START_FROM_CURRENT (com.hazelcast.jet.pipeline.JournalInitialPosition.START_FROM_CURRENT)3 Pipeline (com.hazelcast.jet.pipeline.Pipeline)3