Examples with PubSubInputProperties org.talend.components.pubsub.input.PubSubInputProperties used on opensource projects

Example 1 with PubSubInputProperties

use of org.talend.components.pubsub.input.PubSubInputProperties in project components by Talend.

the class PubSubDatasetRuntime method getSample.

@Override
public void getSample(int limit, Consumer<IndexedRecord> consumer) {
    // Because PubSub do not have offset, and the message will be deleted after
    // read, so have to create a dumy reader which do not call ack after read
    // Create an input runtime based on the properties.
    PubSubInputRuntime inputRuntime = new PubSubInputRuntime();
    PubSubInputProperties inputProperties = new PubSubInputProperties(null);
    inputProperties.init();
    inputProperties.setDatasetProperties(properties);
    inputProperties.useMaxNumRecords.setValue(true);
    inputProperties.maxNumRecords.setValue(limit);
    inputProperties.useMaxReadTime.setValue(true);
    // 10s, the value is better to depends on ack deadline for small dataset
    inputProperties.maxReadTime.setValue(10000l);
    inputProperties.noACK.setValue(true);
    inputRuntime.initialize(null, inputProperties);
    DirectOptions options = BeamLocalRunnerOption.getOptions();
    final Pipeline p = Pipeline.create(options);
    try (DirectConsumerCollector<IndexedRecord> collector = DirectConsumerCollector.of(consumer)) {
        // Collect a sample of the input records.
        // 
        p.apply(inputRuntime).apply(Sample.<IndexedRecord>any(limit)).apply(collector);
        p.run().waitUntilFinish();
    }
}

Example 2 with PubSubInputProperties

use of org.talend.components.pubsub.input.PubSubInputProperties in project components by Talend.

the class PubSubTestConstants method createInput.

public static PubSubInputProperties createInput(PubSubDatasetProperties dataset, Long maxTime, Integer maxNum) {
    PubSubInputProperties input = new PubSubInputProperties("input");
    input.init();
    input.setDatasetProperties(dataset);
    if (maxTime != null) {
        input.useMaxReadTime.setValue(true);
        input.maxReadTime.setValue(maxTime);
    }
    if (maxNum != null) {
        input.useMaxNumRecords.setValue(true);
        input.maxNumRecords.setValue(maxNum);
    }
    return input;
}