Examples with SerializableConfiguration org.apache.beam.sdk.io.hadoop.SerializableConfiguration used on opensource projects

Example 6 with SerializableConfiguration

use of org.apache.beam.sdk.io.hadoop.SerializableConfiguration in project beam by apache.

the class HadoopInputFormatIOTest method testReadBuildsCorrectlyIfWithConfigurationIsCalledMoreThanOneTime.

/**
   * This test validates {@link HadoopInputFormatIO.Read Read} object creation if
   * {@link HadoopInputFormatIO.Read#withConfiguration() withConfiguration()} is called more than
   * once.
   * @throws InterruptedException
   * @throws IOException
   */
@Test
public void testReadBuildsCorrectlyIfWithConfigurationIsCalledMoreThanOneTime() throws IOException, InterruptedException {
    SerializableConfiguration diffConf = loadTestConfiguration(EmployeeInputFormat.class, Employee.class, Text.class);
    HadoopInputFormatIO.Read<String, String> read = HadoopInputFormatIO.<String, String>read().withConfiguration(serConf.get()).withKeyTranslation(myKeyTranslate).withConfiguration(diffConf.get());
    assertEquals(diffConf.get(), read.getConfiguration().get());
    assertEquals(myKeyTranslate, read.getKeyTranslationFunction());
    assertEquals(null, read.getValueTranslationFunction());
    assertEquals(myKeyTranslate.getOutputTypeDescriptor(), read.getKeyTypeDescriptor());
    assertEquals(diffConf.get().getClass("value.class", Object.class), read.getValueTypeDescriptor().getRawType());
}

Example 7 with SerializableConfiguration

use of org.apache.beam.sdk.io.hadoop.SerializableConfiguration in project beam by apache.

the class HadoopFormatIOReadTest method loadTestConfiguration.

private static SerializableConfiguration loadTestConfiguration(Class<?> inputFormatClassName, Class<?> keyClass, Class<?> valueClass) {
    Configuration conf = new Configuration();
    conf.setClass("mapreduce.job.inputformat.class", inputFormatClassName, InputFormat.class);
    conf.setClass("key.class", keyClass, Object.class);
    conf.setClass("value.class", valueClass, Object.class);
    return new SerializableConfiguration(conf);
}