Examples with PCollectionList org.apache.beam.sdk.values.PCollectionList used on opensource projects

Example 1 with PCollectionList

use of org.apache.beam.sdk.values.PCollectionList in project beam by apache.

the class PTransformMatchersTest method emptyFlattenWithNonEmptyFlatten.

@Test
public void emptyFlattenWithNonEmptyFlatten() {
    AppliedPTransform application = AppliedPTransform.<PCollectionList<Object>, PCollection<Object>, Flatten.PCollections<Object>>of("Flatten", Collections.<TupleTag<?>, PValue>singletonMap(new TupleTag<Object>(), PCollection.createPrimitiveOutputInternal(p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED)), Collections.<TupleTag<?>, PValue>singletonMap(new TupleTag<Object>(), PCollection.createPrimitiveOutputInternal(p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED)), Flatten.pCollections(), p);
    assertThat(PTransformMatchers.emptyFlatten().matches(application), is(false));
}

Example 2 with PCollectionList

use of org.apache.beam.sdk.values.PCollectionList in project beam by apache.

the class TransformHierarchyTest method emptyCompositeSucceeds.

@Test
public void emptyCompositeSucceeds() {
    PCollection<Long> created = PCollection.createPrimitiveOutputInternal(pipeline, WindowingStrategy.globalDefault(), IsBounded.BOUNDED);
    TransformHierarchy.Node node = hierarchy.pushNode("Create", PBegin.in(pipeline), Create.of(1));
    hierarchy.setOutput(created);
    hierarchy.popNode();
    PCollectionList<Long> pcList = PCollectionList.of(created);
    TransformHierarchy.Node emptyTransform = hierarchy.pushNode("Extract", pcList, new PTransform<PCollectionList<Long>, PCollection<Long>>() {

        @Override
        public PCollection<Long> expand(PCollectionList<Long> input) {
            return input.get(0);
        }
    });
    hierarchy.setOutput(created);
    hierarchy.popNode();
    assertThat(hierarchy.getProducer(created), equalTo(node));
    assertThat("A Transform that produces non-primitive output should be composite", emptyTransform.isCompositeNode(), is(true));
}

Example 3 with PCollectionList

use of org.apache.beam.sdk.values.PCollectionList in project beam by apache.

the class DeduplicatedFlattenFactoryTest method duplicatesInsertsMultipliers.

@Test
public void duplicatesInsertsMultipliers() {
    PTransform<PCollectionList<String>, PCollection<String>> replacement = new DeduplicatedFlattenFactory.FlattenWithoutDuplicateInputs<>();
    final PCollectionList<String> inputList = PCollectionList.of(first).and(second).and(first).and(first);
    inputList.apply(replacement);
    pipeline.traverseTopologically(new Defaults() {

        @Override
        public void visitPrimitiveTransform(TransformHierarchy.Node node) {
            if (node.getTransform() instanceof Flatten.PCollections) {
                assertThat(node.getInputs(), not(equalTo(inputList.expand())));
            }
        }
    });
}

Example 4 with PCollectionList

use of org.apache.beam.sdk.values.PCollectionList in project beam by apache.

the class PTransformMatchersTest method flattenWithDuplicateInputsWithoutDuplicates.

@Test
public void flattenWithDuplicateInputsWithoutDuplicates() {
    AppliedPTransform application = AppliedPTransform.<PCollectionList<Object>, PCollection<Object>, Flatten.PCollections<Object>>of("Flatten", Collections.<TupleTag<?>, PValue>singletonMap(new TupleTag<Object>(), PCollection.createPrimitiveOutputInternal(p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED)), Collections.<TupleTag<?>, PValue>singletonMap(new TupleTag<Object>(), PCollection.createPrimitiveOutputInternal(p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED)), Flatten.pCollections(), p);
    assertThat(PTransformMatchers.flattenWithDuplicateInputs().matches(application), is(false));
}

Example 5 with PCollectionList

use of org.apache.beam.sdk.values.PCollectionList in project beam by apache.

the class PTransformMatchersTest method emptyFlattenWithEmptyFlatten.

@Test
public void emptyFlattenWithEmptyFlatten() {
    AppliedPTransform application = AppliedPTransform.<PCollectionList<Object>, PCollection<Object>, Flatten.PCollections<Object>>of("EmptyFlatten", Collections.<TupleTag<?>, PValue>emptyMap(), Collections.<TupleTag<?>, PValue>singletonMap(new TupleTag<Object>(), PCollection.createPrimitiveOutputInternal(p, WindowingStrategy.globalDefault(), IsBounded.BOUNDED)), Flatten.pCollections(), p);
    assertThat(PTransformMatchers.emptyFlatten().matches(application), is(true));
}