Examples with DynamicSourceSplit com.google.api.services.dataflow.model.DynamicSourceSplit used on opensource projects

Example 1 with DynamicSourceSplit

use of com.google.api.services.dataflow.model.DynamicSourceSplit in project beam by apache.

the class WorkerCustomSourcesTest method testNegativeEstimatedSizesNotSet.

@Test
public void testNegativeEstimatedSizesNotSet() throws Exception {
    WorkerCustomSources.BoundedSourceSplit<Integer> boundedSourceSplit = new WorkerCustomSources.BoundedSourceSplit<Integer>(new SourceProducingNegativeEstimatedSizes(), new SourceProducingNegativeEstimatedSizes());
    DynamicSourceSplit dynamicSourceSplit = WorkerCustomSources.toSourceSplit(boundedSourceSplit);
    assertNull(dynamicSourceSplit.getPrimary().getSource().getMetadata().getEstimatedSizeBytes());
    assertNull(dynamicSourceSplit.getResidual().getSource().getMetadata().getEstimatedSizeBytes());
}

Example 2 with DynamicSourceSplit

use of com.google.api.services.dataflow.model.DynamicSourceSplit in project beam by apache.

the class WorkerCustomSources method toSourceSplit.

public static DynamicSourceSplit toSourceSplit(BoundedSourceSplit<?> sourceSplitResult) {
    DynamicSourceSplit sourceSplit = new DynamicSourceSplit();
    com.google.api.services.dataflow.model.Source primarySource;
    com.google.api.services.dataflow.model.Source residualSource;
    try {
        primarySource = serializeSplitToCloudSource(sourceSplitResult.primary);
        residualSource = serializeSplitToCloudSource(sourceSplitResult.residual);
    } catch (Exception e) {
        throw new RuntimeException("Failed to serialize one of the parts of the source split", e);
    }
    sourceSplit.setPrimary(new DerivedSource().setDerivationMode("SOURCE_DERIVATION_MODE_INDEPENDENT").setSource(primarySource));
    sourceSplit.setResidual(new DerivedSource().setDerivationMode("SOURCE_DERIVATION_MODE_INDEPENDENT").setSource(residualSource));
    return sourceSplit;
}