Examples with TopicRange org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange used on opensource projects

Example 1 with TopicRange

use of org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange in project flink by apache.

the class PulsarPartitionSplitSerializer method deserializeTopicPartition.

public TopicPartition deserializeTopicPartition(int version, DataInputStream in) throws IOException {
    // VERSION 0 deserialization
    String topic = in.readUTF();
    int partitionId = in.readInt();
    int start = in.readInt();
    int end = in.readInt();
    TopicRange range = new TopicRange(start, end);
    return new TopicPartition(topic, partitionId, range);
}

Example 2 with TopicRange

use of org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange in project flink by apache.

the class PulsarSourceEnumStateSerializerTest method serializeAndDeserializePulsarSourceEnumState.

@Test
void serializeAndDeserializePulsarSourceEnumState() throws Exception {
    Set<TopicPartition> partitions = Sets.newHashSet(new TopicPartition(randomAlphabetic(10), 2, new TopicRange(1, 30)), new TopicPartition(randomAlphabetic(10), 1, createFullRange()));
    Set<PulsarPartitionSplit> splits = Collections.singleton(new PulsarPartitionSplit(new TopicPartition(randomAlphabetic(10), 10, createFullRange()), StopCursor.defaultStopCursor()));
    Map<Integer, Set<PulsarPartitionSplit>> shared = Collections.singletonMap(5, splits);
    Map<Integer, Set<String>> mapping = ImmutableMap.of(1, Sets.newHashSet(randomAlphabetic(10), randomAlphabetic(10)), 2, Sets.newHashSet(randomAlphabetic(10), randomAlphabetic(10)));
    PulsarSourceEnumState state = new PulsarSourceEnumState(partitions, splits, shared, mapping, true);
    byte[] bytes = INSTANCE.serialize(state);
    PulsarSourceEnumState state1 = INSTANCE.deserialize(INSTANCE.getVersion(), bytes);
    assertEquals(state.getAppendedPartitions(), state1.getAppendedPartitions());
    assertEquals(state.getPendingPartitionSplits(), state1.getPendingPartitionSplits());
    assertEquals(state.getReaderAssignedSplits(), state1.getReaderAssignedSplits());
    assertEquals(state.isInitialized(), state1.isInitialized());
    assertNotSame(state, state1);
}

Example 3 with TopicRange

use of org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange in project flink by apache.

the class PulsarPartitionSplitSerializer method serializeTopicPartition.

public void serializeTopicPartition(DataOutputStream out, TopicPartition partition) throws IOException {
    // VERSION 0 serialization
    TopicRange range = partition.getRange();
    out.writeUTF(partition.getTopic());
    out.writeInt(partition.getPartitionId());
    out.writeInt(range.getStart());
    out.writeInt(range.getEnd());
}

Example 4 with TopicRange

use of org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange in project flink by apache.

the class KeySharedSubscriptionContext method createSource.

@Override
public Source<String, ?, ?> createSource(TestingSourceSettings sourceSettings) {
    int keyHash = keyHash(key1);
    TopicRange range = new TopicRange(keyHash, keyHash);
    PulsarSourceBuilder<String> builder = PulsarSource.builder().setDeserializationSchema(pulsarSchema(STRING)).setServiceUrl(operator.serviceUrl()).setAdminUrl(operator.adminUrl()).setTopicPattern("pulsar-[0-9]+-key-shared", RegexSubscriptionMode.AllTopics).setSubscriptionType(SubscriptionType.Key_Shared).setSubscriptionName("pulsar-key-shared").setRangeGenerator(new FixedRangeGenerator(singletonList(range)));
    if (sourceSettings.getBoundedness() == Boundedness.BOUNDED) {
        // Using latest stop cursor for making sure the source could be stopped.
        builder.setBoundedStopCursor(StopCursor.latest());
    }
    return builder.build();
}

Example 5 with TopicRange

use of org.apache.flink.connector.pulsar.source.enumerator.topic.TopicRange in project flink by apache.

the class UniformRangeGenerator method range.

@Override
public List<TopicRange> range(TopicMetadata metadata, int parallelism) {
    List<TopicRange> results = new ArrayList<>(parallelism);
    int startRange = 0;
    for (int i = 1; i < parallelism; i++) {
        int nextStartRange = i * RANGE_SIZE / parallelism;
        results.add(new TopicRange(startRange, nextStartRange - 1));
        startRange = nextStartRange;
    }
    results.add(new TopicRange(startRange, MAX_RANGE));
    return results;
}