Search in sources :

Example 1 with EventHubRuntimeInformation

use of com.microsoft.azure.eventhubs.EventHubRuntimeInformation in project samza by apache.

the class EventHubSystemAdmin method getSystemStreamMetadata.

@Override
public Map<String, SystemStreamMetadata> getSystemStreamMetadata(Set<String> streamNames) {
    Map<String, SystemStreamMetadata> requestedMetadata = new HashMap<>();
    try {
        for (String streamName : streamNames) {
            if (!streamPartitions.containsKey(streamName)) {
                LOG.debug(String.format("Partition ids for Stream=%s not found", streamName));
                EventHubClientManager eventHubClientManager = getOrCreateStreamEventHubClient(streamName);
                EventHubClient ehClient = eventHubClientManager.getEventHubClient();
                CompletableFuture<EventHubRuntimeInformation> runtimeInfo = ehClient.getRuntimeInformation();
                long timeoutMs = eventHubConfig.getRuntimeInfoWaitTimeMS(systemName);
                EventHubRuntimeInformation ehInfo = runtimeInfo.get(timeoutMs, TimeUnit.MILLISECONDS);
                LOG.info(String.format("Adding partition ids=%s for stream=%s. EHRuntimetInfo=%s", Arrays.toString(ehInfo.getPartitionIds()), streamName, printEventHubRuntimeInfo(ehInfo)));
                streamPartitions.put(streamName, ehInfo.getPartitionIds());
            }
            String[] partitionIds = streamPartitions.get(streamName);
            Map<Partition, SystemStreamPartitionMetadata> sspMetadataMap = getPartitionMetadata(streamName, partitionIds);
            SystemStreamMetadata systemStreamMetadata = new SystemStreamMetadata(streamName, sspMetadataMap);
            requestedMetadata.put(streamName, systemStreamMetadata);
        }
        return requestedMetadata;
    } catch (Exception e) {
        String msg = String.format("Error while fetching EventHubRuntimeInfo for System:%s", systemName);
        LOG.error(msg, e);
        throw new SamzaException(msg, e);
    }
}
Also used : SystemStreamPartition(org.apache.samza.system.SystemStreamPartition) Partition(org.apache.samza.Partition) HashMap(java.util.HashMap) SystemStreamMetadata(org.apache.samza.system.SystemStreamMetadata) SystemStreamPartitionMetadata(org.apache.samza.system.SystemStreamMetadata.SystemStreamPartitionMetadata) SamzaException(org.apache.samza.SamzaException) TimeoutException(java.util.concurrent.TimeoutException) EventHubException(com.microsoft.azure.eventhubs.EventHubException) SamzaException(org.apache.samza.SamzaException) ExecutionException(java.util.concurrent.ExecutionException) EventHubRuntimeInformation(com.microsoft.azure.eventhubs.EventHubRuntimeInformation) EventHubClient(com.microsoft.azure.eventhubs.EventHubClient) EventHubClientManager(org.apache.samza.system.eventhub.EventHubClientManager)

Example 2 with EventHubRuntimeInformation

use of com.microsoft.azure.eventhubs.EventHubRuntimeInformation in project samza by apache.

the class EventHubConsoleConsumer method consumeEvents.

private static void consumeEvents(String ehName, String namespace, String keyName, String token) throws EventHubException, IOException, ExecutionException, InterruptedException {
    ConnectionStringBuilder connStr = new ConnectionStringBuilder().setNamespaceName(namespace).setEventHubName(ehName).setSasKeyName(keyName).setSasKey(token);
    EventHubClient client = EventHubClient.createSync(connStr.toString(), Executors.newFixedThreadPool(10));
    EventHubRuntimeInformation runTimeInfo = client.getRuntimeInformation().get();
    int numPartitions = runTimeInfo.getPartitionCount();
    for (int partition = 0; partition < numPartitions; partition++) {
        PartitionReceiver receiver = client.createReceiverSync(EventHubClient.DEFAULT_CONSUMER_GROUP_NAME, String.valueOf(partition), EventPosition.fromStartOfStream());
        receiver.receive(10).handle((records, throwable) -> handleComplete(receiver, records, throwable));
    }
}
Also used : ConnectionStringBuilder(com.microsoft.azure.eventhubs.ConnectionStringBuilder) PartitionReceiver(com.microsoft.azure.eventhubs.PartitionReceiver) EventHubClient(com.microsoft.azure.eventhubs.EventHubClient) EventHubRuntimeInformation(com.microsoft.azure.eventhubs.EventHubRuntimeInformation)

Aggregations

EventHubClient (com.microsoft.azure.eventhubs.EventHubClient)2 EventHubRuntimeInformation (com.microsoft.azure.eventhubs.EventHubRuntimeInformation)2 ConnectionStringBuilder (com.microsoft.azure.eventhubs.ConnectionStringBuilder)1 EventHubException (com.microsoft.azure.eventhubs.EventHubException)1 PartitionReceiver (com.microsoft.azure.eventhubs.PartitionReceiver)1 HashMap (java.util.HashMap)1 ExecutionException (java.util.concurrent.ExecutionException)1 TimeoutException (java.util.concurrent.TimeoutException)1 Partition (org.apache.samza.Partition)1 SamzaException (org.apache.samza.SamzaException)1 SystemStreamMetadata (org.apache.samza.system.SystemStreamMetadata)1 SystemStreamPartitionMetadata (org.apache.samza.system.SystemStreamMetadata.SystemStreamPartitionMetadata)1 SystemStreamPartition (org.apache.samza.system.SystemStreamPartition)1 EventHubClientManager (org.apache.samza.system.eventhub.EventHubClientManager)1