Search in sources :

Example 6 with SerializableConfiguration

use of org.apache.beam.sdk.io.hadoop.SerializableConfiguration in project beam by apache.

the class HadoopInputFormatIOTest method testReadBuildsCorrectlyIfWithConfigurationIsCalledMoreThanOneTime.

/**
   * This test validates {@link HadoopInputFormatIO.Read Read} object creation if
   * {@link HadoopInputFormatIO.Read#withConfiguration() withConfiguration()} is called more than
   * once.
   * @throws InterruptedException
   * @throws IOException
   */
@Test
public void testReadBuildsCorrectlyIfWithConfigurationIsCalledMoreThanOneTime() throws IOException, InterruptedException {
    SerializableConfiguration diffConf = loadTestConfiguration(EmployeeInputFormat.class, Employee.class, Text.class);
    HadoopInputFormatIO.Read<String, String> read = HadoopInputFormatIO.<String, String>read().withConfiguration(serConf.get()).withKeyTranslation(myKeyTranslate).withConfiguration(diffConf.get());
    assertEquals(diffConf.get(), read.getConfiguration().get());
    assertEquals(myKeyTranslate, read.getKeyTranslationFunction());
    assertEquals(null, read.getValueTranslationFunction());
    assertEquals(myKeyTranslate.getOutputTypeDescriptor(), read.getKeyTypeDescriptor());
    assertEquals(diffConf.get().getClass("value.class", Object.class), read.getValueTypeDescriptor().getRawType());
}
Also used : SerializableConfiguration(org.apache.beam.sdk.io.hadoop.SerializableConfiguration) Test(org.junit.Test)

Example 7 with SerializableConfiguration

use of org.apache.beam.sdk.io.hadoop.SerializableConfiguration in project beam by apache.

the class HadoopFormatIOReadTest method loadTestConfiguration.

private static SerializableConfiguration loadTestConfiguration(Class<?> inputFormatClassName, Class<?> keyClass, Class<?> valueClass) {
    Configuration conf = new Configuration();
    conf.setClass("mapreduce.job.inputformat.class", inputFormatClassName, InputFormat.class);
    conf.setClass("key.class", keyClass, Object.class);
    conf.setClass("value.class", valueClass, Object.class);
    return new SerializableConfiguration(conf);
}
Also used : Configuration(org.apache.hadoop.conf.Configuration) SerializableConfiguration(org.apache.beam.sdk.io.hadoop.SerializableConfiguration) SerializableConfiguration(org.apache.beam.sdk.io.hadoop.SerializableConfiguration)

Aggregations

SerializableConfiguration (org.apache.beam.sdk.io.hadoop.SerializableConfiguration)7 Configuration (org.apache.hadoop.conf.Configuration)4 Test (org.junit.Test)4 BoundedSource (org.apache.beam.sdk.io.BoundedSource)1 HadoopInputFormatBoundedSource (org.apache.beam.sdk.io.hadoop.format.HadoopFormatIO.HadoopInputFormatBoundedSource)1 KV (org.apache.beam.sdk.values.KV)1 Text (org.apache.hadoop.io.Text)1 DBConfiguration (org.apache.hadoop.mapreduce.lib.db.DBConfiguration)1