Search in sources :

Example 26 with BatchSourceConfig

use of org.apache.pulsar.common.io.BatchSourceConfig in project pulsar by apache.

the class SourceConfigUtilsTest method createSourceConfigWithBatch.

private SourceConfig createSourceConfigWithBatch() {
    SourceConfig sourceConfig = createSourceConfig();
    BatchSourceConfig batchSourceConfig = createBatchSourceConfig();
    sourceConfig.setBatchSourceConfig(batchSourceConfig);
    return sourceConfig;
}
Also used : BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) SourceConfig(org.apache.pulsar.common.io.SourceConfig)

Example 27 with BatchSourceConfig

use of org.apache.pulsar.common.io.BatchSourceConfig in project pulsar by apache.

the class SourceConfigUtilsTest method testMergeDifferentBatchTriggerer.

@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = "DiscoverTriggerer class cannot be updated for batchsources")
public void testMergeDifferentBatchTriggerer() {
    SourceConfig sourceConfig = createSourceConfigWithBatch();
    BatchSourceConfig batchSourceConfig = createBatchSourceConfig();
    batchSourceConfig.setDiscoveryTriggererClassName("SomeOtherClassName");
    SourceConfig newSourceConfig = createUpdatedSourceConfig("batchSourceConfig", batchSourceConfig);
    SourceConfigUtils.validateUpdate(sourceConfig, newSourceConfig);
}
Also used : BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) SourceConfig(org.apache.pulsar.common.io.SourceConfig) Test(org.testng.annotations.Test)

Example 28 with BatchSourceConfig

use of org.apache.pulsar.common.io.BatchSourceConfig in project pulsar by apache.

the class SourceConfigUtilsTest method createBatchSourceConfig.

private BatchSourceConfig createBatchSourceConfig() {
    BatchSourceConfig batchSourceConfig = new BatchSourceConfig();
    batchSourceConfig.setDiscoveryTriggererClassName(TestTriggerer.class.getName());
    Map<String, Object> batchConfig = new HashMap<>();
    batchConfig.put("foo", "bar");
    batchSourceConfig.setDiscoveryTriggererConfig(batchConfig);
    return batchSourceConfig;
}
Also used : BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) HashMap(java.util.HashMap)

Example 29 with BatchSourceConfig

use of org.apache.pulsar.common.io.BatchSourceConfig in project pulsar by apache.

the class BatchDataGeneratorExec method main.

public static void main(final String[] args) throws Exception {
    final String cronString = "0 0/5 * * * ?";
    final Map<String, Object> discoveryConfig = new HashMap<>();
    discoveryConfig.put(CronTriggerer.CRON_KEY, cronString);
    final BatchSourceConfig batchSourceConfig = BatchSourceConfig.builder().discoveryTriggererClassName(CronTriggerer.class.getName()).discoveryTriggererConfig(discoveryConfig).build();
    final SourceConfig sourceConfig = SourceConfig.builder().batchSourceConfig(batchSourceConfig).className(BatchDataGeneratorSource.class.getName()).configs(new HashMap<>()).name("BatchDataGenerator").parallelism(1).topicName("persistent://public/default/batchdatagenerator").build();
    final LocalRunner localRunner = LocalRunner.builder().brokerServiceUrl("pulsar://localhost:6650").sourceConfig(sourceConfig).build();
    localRunner.start(false);
    TimeUnit.MINUTES.sleep(30);
    localRunner.stop();
    System.exit(0);
}
Also used : BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) CronTriggerer(org.apache.pulsar.io.batchdiscovery.CronTriggerer) LocalRunner(org.apache.pulsar.functions.LocalRunner) HashMap(java.util.HashMap) BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) SourceConfig(org.apache.pulsar.common.io.SourceConfig)

Example 30 with BatchSourceConfig

use of org.apache.pulsar.common.io.BatchSourceConfig in project pulsar by apache.

the class TestCmdSources method testBatchSourceConfigMissingDiscoveryTriggererClassName.

/*
     * Test where the DiscoveryTriggererClassName is null
     */
@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = "Discovery Triggerer not specified")
public void testBatchSourceConfigMissingDiscoveryTriggererClassName() throws Exception {
    SourceConfig testSourceConfig = getSourceConfig();
    BatchSourceConfig batchSourceConfig = getBatchSourceConfig();
    batchSourceConfig.setDiscoveryTriggererClassName(null);
    testSourceConfig.setBatchSourceConfig(batchSourceConfig);
    SourceConfig expectedSourceConfig = getSourceConfig();
    expectedSourceConfig.setBatchSourceConfig(batchSourceConfig);
    testCmdSourceConfigFile(testSourceConfig, expectedSourceConfig);
}
Also used : BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) BatchSourceConfig(org.apache.pulsar.common.io.BatchSourceConfig) SourceConfig(org.apache.pulsar.common.io.SourceConfig) Test(org.testng.annotations.Test)

Aggregations

BatchSourceConfig (org.apache.pulsar.common.io.BatchSourceConfig)30 HashMap (java.util.HashMap)18 SourceConfig (org.apache.pulsar.common.io.SourceConfig)18 Test (org.testng.annotations.Test)9 Gson (com.google.gson.Gson)6 Type (java.lang.reflect.Type)3 Map (java.util.Map)3 ProducerConfig (org.apache.pulsar.common.functions.ProducerConfig)3 Resources (org.apache.pulsar.common.functions.Resources)3 LocalRunner (org.apache.pulsar.functions.LocalRunner)3 IdentityFunction (org.apache.pulsar.functions.api.utils.IdentityFunction)3 Function (org.apache.pulsar.functions.proto.Function)3 FunctionCommon.getSourceType (org.apache.pulsar.functions.utils.FunctionCommon.getSourceType)3 CronTriggerer (org.apache.pulsar.io.batchdiscovery.CronTriggerer)3