Examples with Timestamper com.metamx.tranquility.typeclass.Timestamper used on opensource projects

Example 1 with Timestamper

use of com.metamx.tranquility.typeclass.Timestamper in project storm by apache.

the class SampleDruidBeamFactoryImpl method makeBeam.

@Override
public Beam<Map<String, Object>> makeBeam(Map<?, ?> conf, IMetricsContext metrics) {
    // Your overlord's druid.service
    final String indexService = "druid/overlord";
    // Your overlord's druid.discovery.curator.path
    final String discoveryPath = "/druid/discovery";
    final String dataSource = "test";
    final List<String> dimensions = ImmutableList.of("publisher", "advertiser");
    List<AggregatorFactory> aggregator = ImmutableList.<AggregatorFactory>of(new CountAggregatorFactory("click"));
    // Tranquility needs to be able to extract timestamps from your object type (in this case, Map<String, Object>).
    final Timestamper<Map<String, Object>> timestamper = new Timestamper<Map<String, Object>>() {

        @Override
        public DateTime timestamp(Map<String, Object> theMap) {
            return new DateTime(theMap.get("timestamp"));
        }
    };
    // Tranquility uses ZooKeeper (through Curator) for coordination.
    final CuratorFramework curator = CuratorFrameworkFactory.builder().connectString(// we can use Storm conf to get config values
    (String) conf.get("druid.tranquility.zk.connect")).retryPolicy(new ExponentialBackoffRetry(1000, 20, 30000)).build();
    curator.start();
    // The JSON serialization of your object must have a timestamp field in a format that Druid understands. By default,
    // Druid expects the field to be called "timestamp" and to be an ISO8601 timestamp.
    final TimestampSpec timestampSpec = new TimestampSpec("timestamp", "auto", null);
    // Tranquility needs to be able to serialize your object type to JSON for transmission to Druid. By default this is
    // done with Jackson. If you want to provide an alternate serializer, you can provide your own via ```.objectWriter(...)```.
    // In this case, we won't provide one, so we're just using Jackson.
    final Beam<Map<String, Object>> beam = DruidBeams.builder(timestamper).curator(curator).discoveryPath(discoveryPath).location(DruidLocation.create(indexService, dataSource)).timestampSpec(timestampSpec).rollup(DruidRollup.create(DruidDimensions.specific(dimensions), aggregator, QueryGranularities.MINUTE)).tuning(ClusteredBeamTuning.builder().segmentGranularity(Granularity.HOUR).windowPeriod(new Period("PT10M")).partitions(1).replicants(1).build()).druidBeamConfig(DruidBeamConfig.builder().indexRetryPeriod(new Period("PT10M")).build()).buildBeam();
    return beam;
}