Search in sources :

Example 16 with TestPipeline

use of org.apache.beam.sdk.testing.TestPipeline in project beam by apache.

the class DataflowPTransformMatchersTest method createCombinePerKeyWithSideInputsPipeline.

/**
 * Creates a simple pipeline with a {@link Combine.PerKey} with side inputs.
 */
private static TestPipeline createCombinePerKeyWithSideInputsPipeline() {
    TestPipeline pipeline = TestPipeline.create().enableAbandonedNodeEnforcement(false);
    PCollection<KV<String, Integer>> input = pipeline.apply(Create.of(KV.of("key", 1))).setCoder(KvCoder.of(StringUtf8Coder.of(), VarIntCoder.of()));
    PCollection<String> sideInput = pipeline.apply(Create.of("side input"));
    PCollectionView<String> sideInputView = sideInput.apply(View.asSingleton());
    input.apply(Combine.<String, Integer, Integer>perKey(new SumCombineFnWithContext()).withSideInputs(sideInputView));
    return pipeline;
}
Also used : TestPipeline(org.apache.beam.sdk.testing.TestPipeline) KV(org.apache.beam.sdk.values.KV)

Example 17 with TestPipeline

use of org.apache.beam.sdk.testing.TestPipeline in project beam by apache.

the class PTransformTranslationTest method data.

@Parameters(name = "{index}: {0}")
public static Iterable<ToAndFromProtoSpec> data() {
    // This pipeline exists for construction, not to run any test.
    // TODO: Leaf node with understood payload - i.e. validate payloads
    ToAndFromProtoSpec readLeaf = ToAndFromProtoSpec.leaf(read(TestPipeline.create()));
    ToAndFromProtoSpec readMultipleInAndOut = ToAndFromProtoSpec.leaf(multiMultiParDo(TestPipeline.create()));
    TestPipeline compositeReadPipeline = TestPipeline.create();
    ToAndFromProtoSpec compositeRead = ToAndFromProtoSpec.composite(generateSequence(compositeReadPipeline), ToAndFromProtoSpec.leaf(read(compositeReadPipeline)));
    ToAndFromProtoSpec rawLeafNullSpec = ToAndFromProtoSpec.leaf(rawPTransformWithNullSpec(TestPipeline.create()));
    return ImmutableList.<ToAndFromProtoSpec>builder().add(readLeaf).add(readMultipleInAndOut).add(compositeRead).add(rawLeafNullSpec).build();
}
Also used : TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Parameters(org.junit.runners.Parameterized.Parameters)

Example 18 with TestPipeline

use of org.apache.beam.sdk.testing.TestPipeline in project beam by apache.

the class UnionTest method testBuild_ImplicitName.

@Test
public void testBuild_ImplicitName() {
    final TestPipeline pipeline = TestUtils.createTestPipeline();
    final PCollection<String> left = TestUtils.createMockDataset(pipeline, TypeDescriptors.strings());
    final PCollection<String> right = TestUtils.createMockDataset(pipeline, TypeDescriptors.strings());
    final PCollection<String> unioned = Union.of(left, right).output();
    final Union union = (Union) TestUtils.getProducer(unioned);
    assertFalse(union.getName().isPresent());
}
Also used : TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Test(org.junit.Test)

Example 19 with TestPipeline

use of org.apache.beam.sdk.testing.TestPipeline in project beam by apache.

the class UnionTest method testBuild.

@Test
public void testBuild() {
    final TestPipeline pipeline = TestUtils.createTestPipeline();
    final PCollection<String> left = TestUtils.createMockDataset(pipeline, TypeDescriptors.strings());
    final PCollection<String> right = TestUtils.createMockDataset(pipeline, TypeDescriptors.strings());
    final PCollection<String> unioned = Union.named("Union1").of(left, right).output();
    final Union union = (Union) TestUtils.getProducer(unioned);
    assertTrue(union.getName().isPresent());
    assertEquals("Union1", union.getName().get());
}
Also used : TestPipeline(org.apache.beam.sdk.testing.TestPipeline) Test(org.junit.Test)

Aggregations

TestPipeline (org.apache.beam.sdk.testing.TestPipeline)19 Test (org.junit.Test)11 KV (org.apache.beam.sdk.values.KV)5 PCollection (org.apache.beam.sdk.values.PCollection)3 ByteString (com.google.protobuf.ByteString)2 Pipeline (org.apache.beam.sdk.Pipeline)2 WriteFiles (org.apache.beam.sdk.io.WriteFiles)2 WriteFilesResult (org.apache.beam.sdk.io.WriteFilesResult)2 StorageObject (com.google.api.services.storage.model.StorageObject)1 EqualsTester (com.google.common.testing.EqualsTester)1 StreamingShardedWriteFactory (org.apache.beam.runners.dataflow.DataflowRunner.StreamingShardedWriteFactory)1 StreamingShardedWriteFactory (org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.StreamingShardedWriteFactory)1 EuphoriaOptions (org.apache.beam.sdk.extensions.euphoria.core.translate.EuphoriaOptions)1 PipelineOptions (org.apache.beam.sdk.options.PipelineOptions)1 ReplacementOutput (org.apache.beam.sdk.runners.PTransformOverrideFactory.ReplacementOutput)1 Instant (org.joda.time.Instant)1 Before (org.junit.Before)1 Parameters (org.junit.runners.Parameterized.Parameters)1 AssertResultProperties (org.talend.components.adapter.beam.example.AssertResultProperties)1 AssertResultSink (org.talend.components.adapter.beam.example.AssertResultSink)1