Examples with KafkaSimpleConsumerFactoryImpl com.linkedin.pinot.core.realtime.impl.kafka.KafkaSimpleConsumerFactoryImpl used on opensource projects

Example 1 with KafkaSimpleConsumerFactoryImpl

use of com.linkedin.pinot.core.realtime.impl.kafka.KafkaSimpleConsumerFactoryImpl in project pinot by linkedin.

the class PinotLLCRealtimeSegmentManager method getPartitionOffset.

private long getPartitionOffset(final String topicName, final String bootstrapHosts, final String offsetCriteria, int partitionId) {
    SimpleConsumerWrapper kafkaConsumer = SimpleConsumerWrapper.forPartitionConsumption(new KafkaSimpleConsumerFactoryImpl(), bootstrapHosts, "dummyClientId", topicName, partitionId, KAFKA_PARTITION_OFFSET_FETCH_TIMEOUT_MILLIS);
    final long startOffset;
    try {
        startOffset = kafkaConsumer.fetchPartitionOffset(offsetCriteria, KAFKA_PARTITION_OFFSET_FETCH_TIMEOUT_MILLIS);
    } catch (TimeoutException e) {
        LOGGER.warn("Timed out when fetching partition offsets for topic {} partition {}", topicName, partitionId);
        throw new RuntimeException(e);
    } finally {
        IOUtils.closeQuietly(kafkaConsumer);
    }
    return startOffset;
}

Example 2 with KafkaSimpleConsumerFactoryImpl

use of com.linkedin.pinot.core.realtime.impl.kafka.KafkaSimpleConsumerFactoryImpl in project pinot by linkedin.

the class PinotTableIdealStateBuilder method getPartitionCount.

public static int getPartitionCount(KafkaStreamMetadata kafkaMetadata) {
    String bootstrapHosts = kafkaMetadata.getBootstrapHosts();
    if (bootstrapHosts == null || bootstrapHosts.isEmpty()) {
        throw new RuntimeException("Invalid value for " + Helix.DataSource.Realtime.Kafka.KAFKA_BROKER_LIST);
    }
    SimpleConsumerWrapper consumerWrapper = SimpleConsumerWrapper.forMetadataConsumption(new KafkaSimpleConsumerFactoryImpl(), kafkaMetadata.getBootstrapHosts(), PinotTableIdealStateBuilder.class.getSimpleName() + "-" + kafkaMetadata.getKafkaTopicName(), KAFKA_CONNECTION_TIMEOUT_MILLIS);
    try {
        return consumerWrapper.getPartitionCount(kafkaMetadata.getKafkaTopicName(), /*maxWaitTimeMs=*/
        5000L);
    } finally {
        IOUtils.closeQuietly(consumerWrapper);
    }
}