Search in sources :

Example 76 with DisplayData

use of org.apache.beam.sdk.transforms.display.DisplayData in project beam by apache.

the class ProxyInvocationHandlerTest method testDisplayDataTypes.

@Test
public void testDisplayDataTypes() {
    Instant now = Instant.now();
    TypedOptions options = PipelineOptionsFactory.as(TypedOptions.class);
    options.setInteger(1234);
    options.setTimestamp(now);
    options.setJavaClass(ProxyInvocationHandlerTest.class);
    options.setObject(new Serializable() {

        @Override
        public String toString() {
            return "foobar";
        }
    });
    DisplayData displayData = DisplayData.from(options);
    assertThat(displayData, hasDisplayItem("integer", 1234));
    assertThat(displayData, hasDisplayItem("timestamp", now));
    assertThat(displayData, hasDisplayItem("javaClass", ProxyInvocationHandlerTest.class));
    assertThat(displayData, hasDisplayItem("object", "foobar"));
}
Also used : Serializable(java.io.Serializable) Instant(org.joda.time.Instant) DisplayData(org.apache.beam.sdk.transforms.display.DisplayData) Test(org.junit.Test)

Example 77 with DisplayData

use of org.apache.beam.sdk.transforms.display.DisplayData in project beam by apache.

the class WriteFilesTest method testCustomShardStrategyDisplayData.

@Test
public void testCustomShardStrategyDisplayData() {
    DynamicDestinations<String, Void, String> dynamicDestinations = DynamicFileDestinations.constant(DefaultFilenamePolicy.fromParams(new Params().withBaseFilename(getBaseOutputDirectory().resolve("file", StandardResolveOptions.RESOLVE_FILE)).withShardTemplate("-SS-of-NN")));
    SimpleSink<Void> sink = new SimpleSink<Void>(getBaseOutputDirectory(), dynamicDestinations, Compression.UNCOMPRESSED) {

        @Override
        public void populateDisplayData(DisplayData.Builder builder) {
            builder.add(DisplayData.item("foo", "bar"));
        }
    };
    WriteFiles<String, ?, String> write = WriteFiles.to(sink).withSharding(new PTransform<PCollection<String>, PCollectionView<Integer>>() {

        @Override
        public PCollectionView<Integer> expand(PCollection<String> input) {
            return null;
        }

        @Override
        public void populateDisplayData(DisplayData.Builder builder) {
            builder.add(DisplayData.item("spam", "ham"));
        }
    });
    DisplayData displayData = DisplayData.from(write);
    assertThat(displayData, hasDisplayItem("sink", sink.getClass()));
    assertThat(displayData, includesDisplayDataFor("sink", sink));
    assertThat(displayData, hasDisplayItem("spam", "ham"));
}
Also used : Params(org.apache.beam.sdk.io.DefaultFilenamePolicy.Params) Matchers.containsString(org.hamcrest.Matchers.containsString) PCollection(org.apache.beam.sdk.values.PCollection) PCollectionView(org.apache.beam.sdk.values.PCollectionView) DisplayData(org.apache.beam.sdk.transforms.display.DisplayData) Test(org.junit.Test)

Example 78 with DisplayData

use of org.apache.beam.sdk.transforms.display.DisplayData in project beam by apache.

the class WriteFilesTest method testDisplayData.

@Test
public void testDisplayData() {
    DynamicDestinations<String, Void, String> dynamicDestinations = DynamicFileDestinations.constant(DefaultFilenamePolicy.fromParams(new Params().withBaseFilename(getBaseOutputDirectory().resolve("file", StandardResolveOptions.RESOLVE_FILE)).withShardTemplate("-SS-of-NN")));
    SimpleSink<Void> sink = new SimpleSink<Void>(getBaseOutputDirectory(), dynamicDestinations, Compression.UNCOMPRESSED) {

        @Override
        public void populateDisplayData(DisplayData.Builder builder) {
            builder.add(DisplayData.item("foo", "bar"));
        }
    };
    WriteFiles<String, ?, String> write = WriteFiles.to(sink);
    DisplayData displayData = DisplayData.from(write);
    assertThat(displayData, hasDisplayItem("sink", sink.getClass()));
    assertThat(displayData, includesDisplayDataFor("sink", sink));
}
Also used : Params(org.apache.beam.sdk.io.DefaultFilenamePolicy.Params) Matchers.containsString(org.hamcrest.Matchers.containsString) DisplayData(org.apache.beam.sdk.transforms.display.DisplayData) Test(org.junit.Test)

Example 79 with DisplayData

use of org.apache.beam.sdk.transforms.display.DisplayData in project beam by apache.

the class ProxyInvocationHandlerTest method testDisplayDataIncludedForDisjointInterfaceHierarchies.

@Test
public void testDisplayDataIncludedForDisjointInterfaceHierarchies() {
    FooOptions fooOptions = PipelineOptionsFactory.as(FooOptions.class);
    fooOptions.setFoo("foo");
    BarOptions barOptions = fooOptions.as(BarOptions.class);
    barOptions.setBar("bar");
    DisplayData data = DisplayData.from(barOptions);
    assertThat(data, hasDisplayItem(allOf(hasKey("foo"), hasNamespace(FooOptions.class))));
    assertThat(data, hasDisplayItem(allOf(hasKey("bar"), hasNamespace(BarOptions.class))));
}
Also used : DisplayData(org.apache.beam.sdk.transforms.display.DisplayData) Test(org.junit.Test)

Example 80 with DisplayData

use of org.apache.beam.sdk.transforms.display.DisplayData in project beam by apache.

the class ProxyInvocationHandlerTest method testDisplayDataFromDeserializedJson.

@Test
public void testDisplayDataFromDeserializedJson() throws Exception {
    FooOptions options = PipelineOptionsFactory.as(FooOptions.class);
    options.setFoo("bar");
    DisplayData data = DisplayData.from(options);
    assertThat(data, hasDisplayItem("foo", "bar"));
    FooOptions deserializedOptions = serializeDeserialize(FooOptions.class, options);
    DisplayData dataAfterDeserialization = DisplayData.from(deserializedOptions);
    assertEquals(data, dataAfterDeserialization);
}
Also used : DisplayData(org.apache.beam.sdk.transforms.display.DisplayData) Test(org.junit.Test)

Aggregations

DisplayData (org.apache.beam.sdk.transforms.display.DisplayData)117 Test (org.junit.Test)115 DisplayDataEvaluator (org.apache.beam.sdk.transforms.display.DisplayDataEvaluator)16 PCollection (org.apache.beam.sdk.values.PCollection)11 ByteString (com.google.protobuf.ByteString)9 Category (org.junit.experimental.categories.Category)9 Matchers.containsString (org.hamcrest.Matchers.containsString)6 Duration (org.joda.time.Duration)6 TableRow (com.google.api.services.bigquery.model.TableRow)4 Params (org.apache.beam.sdk.io.DefaultFilenamePolicy.Params)3 BigQueryHelpers.toJsonString (org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.toJsonString)3 Builder (org.apache.beam.sdk.transforms.display.DisplayData.Builder)3 Instant (org.joda.time.Instant)3 TableSchema (com.google.api.services.bigquery.model.TableSchema)2 RowFilter (com.google.bigtable.v2.RowFilter)2 ArrayList (java.util.ArrayList)2 WritableByteChannelFactory (org.apache.beam.sdk.io.FileBasedSink.WritableByteChannelFactory)2 ParDo (org.apache.beam.sdk.transforms.ParDo)2 StringUtils.byteArrayToJsonString (org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString)2 ImmutableSet (com.google.common.collect.ImmutableSet)1