Search in sources :

Example 6 with Auction

use of org.apache.beam.sdk.nexmark.model.Auction in project beam by apache.

the class Query4 method expand.

@Override
public PCollection<CategoryPrice> expand(PCollection<Event> events) {
    PCollection<AuctionBid> winningBids = events.apply(Filter.by(new AuctionOrBid())).apply(new WinningBids(name + ".WinningBids", configuration));
    // Monitor winning bids
    winningBids = winningBids.apply(name + ".WinningBidsMonitor", winningBidsMonitor.getTransform());
    return winningBids.apply(name + ".Rekey", ParDo.of(new DoFn<AuctionBid, KV<Long, Long>>() {

        @ProcessElement
        public void processElement(ProcessContext c) {
            Auction auction = c.element().auction;
            Bid bid = c.element().bid;
            c.output(KV.of(auction.category, bid.price));
        }
    })).apply(Window.into(SlidingWindows.of(Duration.standardSeconds(configuration.windowSizeSec)).every(Duration.standardSeconds(configuration.windowPeriodSec)))).apply(Mean.<Long, Long>perKey().withHotKeyFanout(configuration.fanout)).apply(name + ".Project", ParDo.of(new DoFn<KV<Long, Double>, CategoryPrice>() {

        @ProcessElement
        public void processElement(ProcessContext c) {
            c.output(new CategoryPrice(c.element().getKey(), Math.round(c.element().getValue()), c.pane().isLast()));
        }
    }));
}
Also used : Auction(org.apache.beam.sdk.nexmark.model.Auction) KV(org.apache.beam.sdk.values.KV) DoFn(org.apache.beam.sdk.transforms.DoFn) AuctionBid(org.apache.beam.sdk.nexmark.model.AuctionBid) CategoryPrice(org.apache.beam.sdk.nexmark.model.CategoryPrice) Bid(org.apache.beam.sdk.nexmark.model.Bid) AuctionBid(org.apache.beam.sdk.nexmark.model.AuctionBid)

Example 7 with Auction

use of org.apache.beam.sdk.nexmark.model.Auction in project beam by apache.

the class Query8 method expand.

@Override
public PCollection<IdNameReserve> expand(PCollection<Event> events) {
    // Window and key new people by their id.
    PCollection<KV<Long, Person>> personsById = events.apply(NexmarkQueryUtil.JUST_NEW_PERSONS).apply("Query8.WindowPersons", Window.into(FixedWindows.of(Duration.standardSeconds(configuration.windowSizeSec)))).apply("PersonById", NexmarkQueryUtil.PERSON_BY_ID);
    // Window and key new auctions by their id.
    PCollection<KV<Long, Auction>> auctionsBySeller = events.apply(NexmarkQueryUtil.JUST_NEW_AUCTIONS).apply("Query8.WindowAuctions", Window.into(FixedWindows.of(Duration.standardSeconds(configuration.windowSizeSec)))).apply("AuctionBySeller", NexmarkQueryUtil.AUCTION_BY_SELLER);
    // Join people and auctions and project the person id, name and auction reserve price.
    return KeyedPCollectionTuple.of(NexmarkQueryUtil.PERSON_TAG, personsById).and(NexmarkQueryUtil.AUCTION_TAG, auctionsBySeller).apply(CoGroupByKey.create()).apply(name + ".Select", ParDo.of(new DoFn<KV<Long, CoGbkResult>, IdNameReserve>() {

        @ProcessElement
        public void processElement(ProcessContext c) {
            @Nullable Person person = c.element().getValue().getOnly(NexmarkQueryUtil.PERSON_TAG, null);
            if (person == null) {
                // Person was not created in last window period.
                return;
            }
            for (Auction auction : c.element().getValue().getAll(NexmarkQueryUtil.AUCTION_TAG)) {
                c.output(new IdNameReserve(person.id, person.name, auction.reserve));
            }
        }
    }));
}
Also used : DoFn(org.apache.beam.sdk.transforms.DoFn) Auction(org.apache.beam.sdk.nexmark.model.Auction) KV(org.apache.beam.sdk.values.KV) IdNameReserve(org.apache.beam.sdk.nexmark.model.IdNameReserve) Person(org.apache.beam.sdk.nexmark.model.Person) Nullable(org.checkerframework.checker.nullness.qual.Nullable) CoGbkResult(org.apache.beam.sdk.transforms.join.CoGbkResult)

Aggregations

Auction (org.apache.beam.sdk.nexmark.model.Auction)7 AuctionBid (org.apache.beam.sdk.nexmark.model.AuctionBid)4 Bid (org.apache.beam.sdk.nexmark.model.Bid)4 DoFn (org.apache.beam.sdk.transforms.DoFn)4 KV (org.apache.beam.sdk.values.KV)4 Nullable (org.checkerframework.checker.nullness.qual.Nullable)3 Person (org.apache.beam.sdk.nexmark.model.Person)2 CoGbkResult (org.apache.beam.sdk.transforms.join.CoGbkResult)2 Instant (org.joda.time.Instant)2 ArrayList (java.util.ArrayList)1 List (java.util.List)1 Map (java.util.Map)1 TreeMap (java.util.TreeMap)1 Counter (org.apache.beam.sdk.metrics.Counter)1 CategoryPrice (org.apache.beam.sdk.nexmark.model.CategoryPrice)1 Event (org.apache.beam.sdk.nexmark.model.Event)1 IdNameReserve (org.apache.beam.sdk.nexmark.model.IdNameReserve)1 NameCityStateId (org.apache.beam.sdk.nexmark.model.NameCityStateId)1 StringsGenerator.nextString (org.apache.beam.sdk.nexmark.sources.generator.model.StringsGenerator.nextString)1