Examples with FileBasedIOTestPipelineOptions org.apache.beam.sdk.io.common.FileBasedIOTestPipelineOptions used on opensource projects

Example 1 with FileBasedIOTestPipelineOptions

use of org.apache.beam.sdk.io.common.FileBasedIOTestPipelineOptions in project beam by apache.

the class AvroIOIT method setup.

@BeforeClass
public static void setup() {
    FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
    filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
    datasetSize = options.getDatasetSize();
    expectedHash = options.getExpectedHash();
    numberOfTextLines = options.getNumberOfRecords();
    settings = InfluxDBSettings.builder().withHost(options.getInfluxHost()).withDatabase(options.getInfluxDatabase()).withMeasurement(options.getInfluxMeasurement()).get();
}

Example 2 with FileBasedIOTestPipelineOptions

use of org.apache.beam.sdk.io.common.FileBasedIOTestPipelineOptions in project beam by apache.

the class TextIOIT method setup.

@BeforeClass
public static void setup() {
    FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
    datasetSize = options.getDatasetSize();
    expectedHash = options.getExpectedHash();
    numberOfTextLines = options.getNumberOfRecords();
    compressionType = Compression.valueOf(options.getCompressionType());
    filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
    numShards = options.getNumberOfShards();
    gatherGcsPerformanceMetrics = options.getReportGcsPerformanceMetrics();
    settings = InfluxDBSettings.builder().withHost(options.getInfluxHost()).withDatabase(options.getInfluxDatabase()).withMeasurement(options.getInfluxMeasurement()).get();
}

Example 3 with FileBasedIOTestPipelineOptions

use of org.apache.beam.sdk.io.common.FileBasedIOTestPipelineOptions in project beam by apache.

the class ParquetIOIT method setup.

@BeforeClass
public static void setup() {
    FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
    numberOfTextLines = options.getNumberOfRecords();
    datasetSize = options.getDatasetSize();
    expectedHash = options.getExpectedHash();
    filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
    settings = InfluxDBSettings.builder().withHost(options.getInfluxHost()).withDatabase(options.getInfluxDatabase()).withMeasurement(options.getInfluxMeasurement()).get();
}

Example 4 with FileBasedIOTestPipelineOptions

use of org.apache.beam.sdk.io.common.FileBasedIOTestPipelineOptions in project beam by apache.

the class TFRecordIOIT method setup.

@BeforeClass
public static void setup() {
    FileBasedIOTestPipelineOptions options = readFileBasedIOITPipelineOptions();
    datasetSize = options.getDatasetSize();
    expectedHash = options.getExpectedHash();
    numberOfTextLines = options.getNumberOfRecords();
    compressionType = Compression.valueOf(options.getCompressionType());
    filenamePrefix = appendTimestampSuffix(options.getFilenamePrefix());
    settings = InfluxDBSettings.builder().withHost(options.getInfluxHost()).withDatabase(options.getInfluxDatabase()).withMeasurement(options.getInfluxMeasurement()).get();
}