Search in sources :

Example 61 with MutableDateTime

use of org.joda.time.MutableDateTime in project joda-time by JodaOrg.

the class TestDateTimeFormatter method testParseInto_monthOnly_parseStartYear.

public void testParseInto_monthOnly_parseStartYear() {
    DateTimeFormatter f = DateTimeFormat.forPattern("M").withLocale(Locale.UK);
    MutableDateTime result = new MutableDateTime(2004, 2, 1, 12, 20, 30, 0, TOKYO);
    assertEquals(1, f.parseInto(result, "1", 0));
    assertEquals(new MutableDateTime(2004, 1, 1, 12, 20, 30, 0, TOKYO), result);
}
Also used : MutableDateTime(org.joda.time.MutableDateTime)

Example 62 with MutableDateTime

use of org.joda.time.MutableDateTime in project joda-time by JodaOrg.

the class TestDateTimeFormatter method testParseInto_monthDay_feb29_newYork_startOfYear.

public void testParseInto_monthDay_feb29_newYork_startOfYear() {
    DateTimeFormatter f = DateTimeFormat.forPattern("M d").withLocale(Locale.UK);
    MutableDateTime result = new MutableDateTime(2004, 1, 1, 0, 0, 0, 0, NEWYORK);
    assertEquals(4, f.parseInto(result, "2 29", 0));
    assertEquals(new MutableDateTime(2004, 2, 29, 0, 0, 0, 0, NEWYORK), result);
}
Also used : MutableDateTime(org.joda.time.MutableDateTime)

Example 63 with MutableDateTime

use of org.joda.time.MutableDateTime in project beam by apache.

the class ParDoTest method testSideInputsWithMultipleWindows.

@Test
@Category(ValidatesRunner.class)
public void testSideInputsWithMultipleWindows() {
    // Tests that the runner can safely run a DoFn that uses side inputs
    // on an input where the element is in multiple windows. The complication is
    // that side inputs are per-window, so the runner has to make sure
    // to process each window individually.
    MutableDateTime mutableNow = Instant.now().toMutableDateTime();
    mutableNow.setMillisOfSecond(0);
    Instant now = mutableNow.toInstant();
    SlidingWindows windowFn = SlidingWindows.of(Duration.standardSeconds(5)).every(Duration.standardSeconds(1));
    PCollectionView<Integer> view = pipeline.apply(Create.of(1)).apply(View.<Integer>asSingleton());
    PCollection<String> res = pipeline.apply(Create.timestamped(TimestampedValue.of("a", now))).apply(Window.<String>into(windowFn)).apply(ParDo.of(new FnWithSideInputs(view)).withSideInputs(view));
    for (int i = 0; i < 4; ++i) {
        Instant base = now.minus(Duration.standardSeconds(i));
        IntervalWindow window = new IntervalWindow(base, base.plus(Duration.standardSeconds(5)));
        PAssert.that(res).inWindow(window).containsInAnyOrder("a:1");
    }
    pipeline.run();
}
Also used : Instant(org.joda.time.Instant) MutableDateTime(org.joda.time.MutableDateTime) StringUtils.byteArrayToJsonString(org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString) Matchers.containsString(org.hamcrest.Matchers.containsString) IntervalWindow(org.apache.beam.sdk.transforms.windowing.IntervalWindow) SlidingWindows(org.apache.beam.sdk.transforms.windowing.SlidingWindows) Category(org.junit.experimental.categories.Category) Test(org.junit.Test)

Example 64 with MutableDateTime

use of org.joda.time.MutableDateTime in project beam by apache.

the class SplittableDoFnTest method testPairWithIndexWindowedTimestamped.

@Test
@Category({ ValidatesRunner.class, UsesSplittableParDo.class })
public void testPairWithIndexWindowedTimestamped() {
    // Tests that Splittable DoFn correctly propagates windowing strategy, windows and timestamps
    // of elements in the input collection.
    MutableDateTime mutableNow = Instant.now().toMutableDateTime();
    mutableNow.setMillisOfSecond(0);
    Instant now = mutableNow.toInstant();
    Instant nowP1 = now.plus(Duration.standardSeconds(1));
    Instant nowP2 = now.plus(Duration.standardSeconds(2));
    SlidingWindows windowFn = SlidingWindows.of(Duration.standardSeconds(5)).every(Duration.standardSeconds(1));
    PCollection<KV<String, Integer>> res = p.apply(Create.timestamped(TimestampedValue.of("a", now), TimestampedValue.of("bb", nowP1), TimestampedValue.of("ccccc", nowP2))).apply(Window.<String>into(windowFn)).apply(ParDo.of(new PairStringWithIndexToLength())).setCoder(KvCoder.of(StringUtf8Coder.of(), BigEndianIntegerCoder.of()));
    assertEquals(windowFn, res.getWindowingStrategy().getWindowFn());
    PCollection<TimestampedValue<KV<String, Integer>>> timestamped = res.apply("Reify timestamps", ParDo.of(new ReifyTimestampsFn<KV<String, Integer>>()));
    for (int i = 0; i < 4; ++i) {
        Instant base = now.minus(Duration.standardSeconds(i));
        IntervalWindow window = new IntervalWindow(base, base.plus(Duration.standardSeconds(5)));
        List<TimestampedValue<KV<String, Integer>>> expectedUnfiltered = Arrays.asList(TimestampedValue.of(KV.of("a", 0), now), TimestampedValue.of(KV.of("bb", 0), nowP1), TimestampedValue.of(KV.of("bb", 1), nowP1), TimestampedValue.of(KV.of("ccccc", 0), nowP2), TimestampedValue.of(KV.of("ccccc", 1), nowP2), TimestampedValue.of(KV.of("ccccc", 2), nowP2), TimestampedValue.of(KV.of("ccccc", 3), nowP2), TimestampedValue.of(KV.of("ccccc", 4), nowP2));
        List<TimestampedValue<KV<String, Integer>>> expected = new ArrayList<>();
        for (TimestampedValue<KV<String, Integer>> tv : expectedUnfiltered) {
            if (!window.start().isAfter(tv.getTimestamp()) && !tv.getTimestamp().isAfter(window.maxTimestamp())) {
                expected.add(tv);
            }
        }
        assertFalse(expected.isEmpty());
        PAssert.that(timestamped).inWindow(window).containsInAnyOrder(expected);
    }
    p.run();
}
Also used : Instant(org.joda.time.Instant) ArrayList(java.util.ArrayList) MutableDateTime(org.joda.time.MutableDateTime) KV(org.apache.beam.sdk.values.KV) TimestampedValue(org.apache.beam.sdk.values.TimestampedValue) IntervalWindow(org.apache.beam.sdk.transforms.windowing.IntervalWindow) SlidingWindows(org.apache.beam.sdk.transforms.windowing.SlidingWindows) Category(org.junit.experimental.categories.Category) Test(org.junit.Test)

Aggregations

MutableDateTime (org.joda.time.MutableDateTime)64 DateTime (org.joda.time.DateTime)8 Chronology (org.joda.time.Chronology)4 Instant (org.joda.time.Instant)4 DateTimeFormatter (org.joda.time.format.DateTimeFormatter)4 ISOChronology (org.joda.time.chrono.ISOChronology)3 IntervalWindow (org.apache.beam.sdk.transforms.windowing.IntervalWindow)2 SlidingWindows (org.apache.beam.sdk.transforms.windowing.SlidingWindows)2 ElasticsearchParseException (org.elasticsearch.ElasticsearchParseException)2 Matchers.containsString (org.hamcrest.Matchers.containsString)2 DateTimeZone (org.joda.time.DateTimeZone)2 ReadableInstant (org.joda.time.ReadableInstant)2 Test (org.junit.Test)2 Category (org.junit.experimental.categories.Category)2 ImmutableMap (com.google.common.collect.ImmutableMap)1 LifecycleStart (io.druid.java.util.common.lifecycle.LifecycleStart)1 FileNotFoundException (java.io.FileNotFoundException)1 IOException (java.io.IOException)1 BigDecimal (java.math.BigDecimal)1 Timestamp (java.sql.Timestamp)1