Examples with SamzaPipelineOptions org.apache.beam.runners.samza.SamzaPipelineOptions used on opensource projects

Example 21 with SamzaPipelineOptions

use of org.apache.beam.runners.samza.SamzaPipelineOptions in project beam by apache.

the class ConfigGeneratorTest method testSamzaYarnExecutionEnvironmentConfig.

@Test
public void testSamzaYarnExecutionEnvironmentConfig() {
    final String yarnPackagePath = "yarn.package.path";
    SamzaPipelineOptions options = PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
    options.setJobName("TestEnvConfig");
    options.setRunner(SamzaRunner.class);
    options.setSamzaExecutionEnvironment(SamzaExecutionEnvironment.YARN);
    options.setConfigOverride(ImmutableMap.<String, String>builder().put(yarnPackagePath, "file://${basedir}/target/${project.artifactId}-${pom.version}-dist.tar.gz").build());
    Pipeline pipeline = Pipeline.create(options);
    pipeline.apply(Create.of(1, 2, 3)).apply(Sum.integersGlobally());
    pipeline.replaceAll(SamzaTransformOverrides.getDefaultOverrides());
    final Map<PValue, String> idMap = PViewToIdMapper.buildIdMap(pipeline);
    final ConfigBuilder configBuilder = new ConfigBuilder(options);
    SamzaPipelineTranslator.createConfig(pipeline, options, idMap, configBuilder);
    try {
        Config config = configBuilder.build();
        assertEquals(config.get(APP_RUNNER_CLASS), RemoteApplicationRunner.class.getName());
        assertEquals(config.get(JOB_FACTORY_CLASS), YarnJobFactory.class.getName());
    } catch (IllegalArgumentException e) {
        throw new AssertionError(String.format("Failed to validate correct configs for %s samza execution environment", SamzaExecutionEnvironment.YARN), e);
    }
}

Example 22 with SamzaPipelineOptions

use of org.apache.beam.runners.samza.SamzaPipelineOptions in project beam by apache.

the class PipelineJsonRendererTest method testEmptyPipeline.

@Test
public void testEmptyPipeline() {
    SamzaPipelineOptions options = PipelineOptionsFactory.create().as(SamzaPipelineOptions.class);
    options.setRunner(SamzaRunner.class);
    Pipeline p = Pipeline.create(options);
    String jsonDag = "{  \"RootNode\": [" + "    { \"fullName\":\"OuterMostNode\"," + "      \"ChildNodes\":[    ]}],\"graphLinks\": []" + "}";
    System.out.println(PipelineJsonRenderer.toJsonString(p));
    assertEquals(JsonParser.parseString(jsonDag), JsonParser.parseString(PipelineJsonRenderer.toJsonString(p).replaceAll(System.lineSeparator(), "")));
}

Example 23 with SamzaPipelineOptions

use of org.apache.beam.runners.samza.SamzaPipelineOptions in project beam by apache.

the class UnboundedSourceSystemTest method createConsumer.

private static UnboundedSourceSystem.Consumer<String, TestCheckpointMark> createConsumer(TestUnboundedSource<String> source, int splitNum) {
    SamzaPipelineOptions pipelineOptions = PipelineOptionsFactory.as(SamzaPipelineOptions.class);
    // emit immediately
    pipelineOptions.setWatermarkInterval(0L);
    pipelineOptions.setMaxSourceParallelism(splitNum);
    return new UnboundedSourceSystem.Consumer<>(source, pipelineOptions, new SamzaMetricsContainer(new MetricsRegistryMap()), "test-step");
}

Example 24 with SamzaPipelineOptions

use of org.apache.beam.runners.samza.SamzaPipelineOptions in project beam by apache.

the class BoundedSourceSystemTest method createConsumer.

private static BoundedSourceSystem.Consumer<String> createConsumer(BoundedSource<String> source, int splitNum) {
    SamzaPipelineOptions pipelineOptions = PipelineOptionsFactory.as(SamzaPipelineOptions.class);
    pipelineOptions.setMaxSourceParallelism(splitNum);
    return new BoundedSourceSystem.Consumer<>(source, pipelineOptions, new SamzaMetricsContainer(new MetricsRegistryMap()), "test-step");
}