Examples with HostInfo org.apache.kafka.streams.state.HostInfo used on opensource projects

Example 31 with HostInfo

use of org.apache.kafka.streams.state.HostInfo in project kafka-streams-examples by confluentinc.

the class KafkaMusicExample method main.

public static void main(String[] args) throws Exception {
    if (args.length == 0 || args.length > 4) {
        throw new IllegalArgumentException("usage: ... <portForRestEndpoint> " + "[<bootstrap.servers> (optional, default: " + DEFAULT_BOOTSTRAP_SERVERS + ")] " + "[<schema.registry.url> (optional, default: " + DEFAULT_SCHEMA_REGISTRY_URL + ")] " + "[<hostnameForRestEndPoint> (optional, default: " + DEFAULT_REST_ENDPOINT_HOSTNAME + ")]");
    }
    final int restEndpointPort = Integer.valueOf(args[0]);
    final String bootstrapServers = args.length > 1 ? args[1] : "localhost:9092";
    final String schemaRegistryUrl = args.length > 2 ? args[2] : "http://localhost:8081";
    final String restEndpointHostname = args.length > 3 ? args[3] : DEFAULT_REST_ENDPOINT_HOSTNAME;
    final HostInfo restEndpoint = new HostInfo(restEndpointHostname, restEndpointPort);
    System.out.println("Connecting to Kafka cluster via bootstrap servers " + bootstrapServers);
    System.out.println("Connecting to Confluent schema registry at " + schemaRegistryUrl);
    System.out.println("REST endpoint at http://" + restEndpointHostname + ":" + restEndpointPort);
    final KafkaStreams streams = createChartsStreams(bootstrapServers, schemaRegistryUrl, restEndpointPort, "/tmp/kafka-streams");
    // Always (and unconditionally) clean local state prior to starting the processing topology.
    // We opt for this unconditional call here because this will make it easier for you to play around with the example
    // when resetting the application for doing a re-run (via the Application Reset Tool,
    // http://docs.confluent.io/current/streams/developer-guide.html#application-reset-tool).
    // 
    // The drawback of cleaning up local state prior is that your app must rebuilt its local state from scratch, which
    // will take time and will require reading all the state-relevant data from the Kafka cluster over the network.
    // Thus in a production scenario you typically do not want to clean up always as we do here but rather only when it
    // is truly needed, i.e., only under certain conditions (e.g., the presence of a command line flag for your app).
    // See `ApplicationResetExample.java` for a production-like example.
    streams.cleanUp();
    // Now that we have finished the definition of the processing topology we can actually run
    // it via `start()`.  The Streams application as a whole can be launched just like any
    // normal Java application that has a `main()` method.
    streams.start();
    // Start the Restful proxy for servicing remote access to state stores
    final MusicPlaysRestService restService = startRestProxy(streams, restEndpoint);
    // Add shutdown hook to respond to SIGTERM and gracefully close Kafka Streams
    Runtime.getRuntime().addShutdownHook(new Thread(() -> {
        try {
            restService.stop();
            streams.close();
        } catch (Exception e) {
        // ignored
        }
    }));
}

Example 32 with HostInfo

use of org.apache.kafka.streams.state.HostInfo in project apache-kafka-on-k8s by banzaicloud.

the class StreamsMetadataState method rebuildMetadata.

private void rebuildMetadata(final Map<HostInfo, Set<TopicPartition>> currentState) {
    allMetadata.clear();
    if (currentState.isEmpty()) {
        return;
    }
    final Map<String, List<String>> stores = builder.stateStoreNameToSourceTopics();
    for (Map.Entry<HostInfo, Set<TopicPartition>> entry : currentState.entrySet()) {
        final HostInfo key = entry.getKey();
        final Set<TopicPartition> partitionsForHost = new HashSet<>(entry.getValue());
        final Set<String> storesOnHost = new HashSet<>();
        for (Map.Entry<String, List<String>> storeTopicEntry : stores.entrySet()) {
            final List<String> topicsForStore = storeTopicEntry.getValue();
            if (hasPartitionsForAnyTopics(topicsForStore, partitionsForHost)) {
                storesOnHost.add(storeTopicEntry.getKey());
            }
        }
        storesOnHost.addAll(globalStores);
        final StreamsMetadata metadata = new StreamsMetadata(key, storesOnHost, partitionsForHost);
        allMetadata.add(metadata);
        if (key.equals(thisHost)) {
            myMetadata = metadata;
        }
    }
}

Example 33 with HostInfo

use of org.apache.kafka.streams.state.HostInfo in project apache-kafka-on-k8s by banzaicloud.

the class StreamsPartitionAssignor method assign.

/*
     * This assigns tasks to consumer clients in the following steps.
     *
     * 0. check all repartition source topics and use internal topic manager to make sure
     *    they have been created with the right number of partitions.
     *
     * 1. using user customized partition grouper to generate tasks along with their
     *    assigned partitions; also make sure that the task's corresponding changelog topics
     *    have been created with the right number of partitions.
     *
     * 2. using TaskAssignor to assign tasks to consumer clients.
     *    - Assign a task to a client which was running it previously.
     *      If there is no such client, assign a task to a client which has its valid local state.
     *    - A client may have more than one stream threads.
     *      The assignor tries to assign tasks to a client proportionally to the number of threads.
     *    - We try not to assign the same set of tasks to two different clients
     *    We do the assignment in one-pass. The result may not satisfy above all.
     *
     * 3. within each client, tasks are assigned to consumer clients in round-robin manner.
     */
@Override
public Map<String, Assignment> assign(final Cluster metadata, final Map<String, Subscription> subscriptions) {
    // construct the client metadata from the decoded subscription info
    final Map<UUID, ClientMetadata> clientsMetadata = new HashMap<>();
    int minUserMetadataVersion = SubscriptionInfo.LATEST_SUPPORTED_VERSION;
    for (final Map.Entry<String, Subscription> entry : subscriptions.entrySet()) {
        final String consumerId = entry.getKey();
        final Subscription subscription = entry.getValue();
        final SubscriptionInfo info = SubscriptionInfo.decode(subscription.userData());
        final int usedVersion = info.version();
        if (usedVersion > SubscriptionInfo.LATEST_SUPPORTED_VERSION) {
            throw new IllegalStateException("Unknown metadata version: " + usedVersion + "; latest supported version: " + SubscriptionInfo.LATEST_SUPPORTED_VERSION);
        }
        if (usedVersion < minUserMetadataVersion) {
            minUserMetadataVersion = usedVersion;
        }
        // create the new client metadata if necessary
        ClientMetadata clientMetadata = clientsMetadata.get(info.processId());
        if (clientMetadata == null) {
            clientMetadata = new ClientMetadata(info.userEndPoint());
            clientsMetadata.put(info.processId(), clientMetadata);
        }
        // add the consumer to the client
        clientMetadata.addConsumer(consumerId, info);
    }
    log.debug("Constructed client metadata {} from the member subscriptions.", clientsMetadata);
    // ---------------- Step Zero ---------------- //
    // parse the topology to determine the repartition source topics,
    // making sure they are created with the number of partitions as
    // the maximum of the depending sub-topologies source topics' number of partitions
    final Map<Integer, InternalTopologyBuilder.TopicsInfo> topicGroups = taskManager.builder().topicGroups();
    final Map<String, InternalTopicMetadata> repartitionTopicMetadata = new HashMap<>();
    for (final InternalTopologyBuilder.TopicsInfo topicsInfo : topicGroups.values()) {
        for (final InternalTopicConfig topic : topicsInfo.repartitionSourceTopics.values()) {
            repartitionTopicMetadata.put(topic.name(), new InternalTopicMetadata(topic));
        }
    }
    boolean numPartitionsNeeded;
    do {
        numPartitionsNeeded = false;
        for (final InternalTopologyBuilder.TopicsInfo topicsInfo : topicGroups.values()) {
            for (final String topicName : topicsInfo.repartitionSourceTopics.keySet()) {
                int numPartitions = repartitionTopicMetadata.get(topicName).numPartitions;
                // try set the number of partitions for this repartition topic if it is not set yet
                if (numPartitions == UNKNOWN) {
                    for (final InternalTopologyBuilder.TopicsInfo otherTopicsInfo : topicGroups.values()) {
                        final Set<String> otherSinkTopics = otherTopicsInfo.sinkTopics;
                        if (otherSinkTopics.contains(topicName)) {
                            // use the maximum of all its source topic partitions as the number of partitions
                            for (final String sourceTopicName : otherTopicsInfo.sourceTopics) {
                                final Integer numPartitionsCandidate;
                                // map().join().join(map())
                                if (repartitionTopicMetadata.containsKey(sourceTopicName)) {
                                    numPartitionsCandidate = repartitionTopicMetadata.get(sourceTopicName).numPartitions;
                                } else {
                                    numPartitionsCandidate = metadata.partitionCountForTopic(sourceTopicName);
                                    if (numPartitionsCandidate == null) {
                                        repartitionTopicMetadata.get(topicName).numPartitions = NOT_AVAILABLE;
                                    }
                                }
                                if (numPartitionsCandidate != null && numPartitionsCandidate > numPartitions) {
                                    numPartitions = numPartitionsCandidate;
                                }
                            }
                        }
                    }
                    // another iteration is needed
                    if (numPartitions == UNKNOWN) {
                        numPartitionsNeeded = true;
                    } else {
                        repartitionTopicMetadata.get(topicName).numPartitions = numPartitions;
                    }
                }
            }
        }
    } while (numPartitionsNeeded);
    // ensure the co-partitioning topics within the group have the same number of partitions,
    // and enforce the number of partitions for those repartition topics to be the same if they
    // are co-partitioned as well.
    ensureCopartitioning(taskManager.builder().copartitionGroups(), repartitionTopicMetadata, metadata);
    // make sure the repartition source topics exist with the right number of partitions,
    // create these topics if necessary
    prepareTopic(repartitionTopicMetadata);
    // augment the metadata with the newly computed number of partitions for all the
    // repartition source topics
    final Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions = new HashMap<>();
    for (final Map.Entry<String, InternalTopicMetadata> entry : repartitionTopicMetadata.entrySet()) {
        final String topic = entry.getKey();
        final int numPartitions = entry.getValue().numPartitions;
        for (int partition = 0; partition < numPartitions; partition++) {
            allRepartitionTopicPartitions.put(new TopicPartition(topic, partition), new PartitionInfo(topic, partition, null, new Node[0], new Node[0]));
        }
    }
    final Cluster fullMetadata = metadata.withPartitions(allRepartitionTopicPartitions);
    taskManager.setClusterMetadata(fullMetadata);
    log.debug("Created repartition topics {} from the parsed topology.", allRepartitionTopicPartitions.values());
    // ---------------- Step One ---------------- //
    // get the tasks as partition groups from the partition grouper
    final Set<String> allSourceTopics = new HashSet<>();
    final Map<Integer, Set<String>> sourceTopicsByGroup = new HashMap<>();
    for (final Map.Entry<Integer, InternalTopologyBuilder.TopicsInfo> entry : topicGroups.entrySet()) {
        allSourceTopics.addAll(entry.getValue().sourceTopics);
        sourceTopicsByGroup.put(entry.getKey(), entry.getValue().sourceTopics);
    }
    final Map<TaskId, Set<TopicPartition>> partitionsForTask = partitionGrouper.partitionGroups(sourceTopicsByGroup, fullMetadata);
    // check if all partitions are assigned, and there are no duplicates of partitions in multiple tasks
    final Set<TopicPartition> allAssignedPartitions = new HashSet<>();
    final Map<Integer, Set<TaskId>> tasksByTopicGroup = new HashMap<>();
    for (final Map.Entry<TaskId, Set<TopicPartition>> entry : partitionsForTask.entrySet()) {
        final Set<TopicPartition> partitions = entry.getValue();
        for (final TopicPartition partition : partitions) {
            if (allAssignedPartitions.contains(partition)) {
                log.warn("Partition {} is assigned to more than one tasks: {}", partition, partitionsForTask);
            }
        }
        allAssignedPartitions.addAll(partitions);
        final TaskId id = entry.getKey();
        Set<TaskId> ids = tasksByTopicGroup.get(id.topicGroupId);
        if (ids == null) {
            ids = new HashSet<>();
            tasksByTopicGroup.put(id.topicGroupId, ids);
        }
        ids.add(id);
    }
    for (final String topic : allSourceTopics) {
        final List<PartitionInfo> partitionInfoList = fullMetadata.partitionsForTopic(topic);
        if (!partitionInfoList.isEmpty()) {
            for (final PartitionInfo partitionInfo : partitionInfoList) {
                final TopicPartition partition = new TopicPartition(partitionInfo.topic(), partitionInfo.partition());
                if (!allAssignedPartitions.contains(partition)) {
                    log.warn("Partition {} is not assigned to any tasks: {}", partition, partitionsForTask);
                }
            }
        } else {
            log.warn("No partitions found for topic {}", topic);
        }
    }
    // add tasks to state change log topic subscribers
    final Map<String, InternalTopicMetadata> changelogTopicMetadata = new HashMap<>();
    for (final Map.Entry<Integer, InternalTopologyBuilder.TopicsInfo> entry : topicGroups.entrySet()) {
        final int topicGroupId = entry.getKey();
        final Map<String, InternalTopicConfig> stateChangelogTopics = entry.getValue().stateChangelogTopics;
        for (final InternalTopicConfig topicConfig : stateChangelogTopics.values()) {
            // the expected number of partitions is the max value of TaskId.partition + 1
            int numPartitions = UNKNOWN;
            if (tasksByTopicGroup.get(topicGroupId) != null) {
                for (final TaskId task : tasksByTopicGroup.get(topicGroupId)) {
                    if (numPartitions < task.partition + 1)
                        numPartitions = task.partition + 1;
                }
                final InternalTopicMetadata topicMetadata = new InternalTopicMetadata(topicConfig);
                topicMetadata.numPartitions = numPartitions;
                changelogTopicMetadata.put(topicConfig.name(), topicMetadata);
            } else {
                log.debug("No tasks found for topic group {}", topicGroupId);
            }
        }
    }
    prepareTopic(changelogTopicMetadata);
    log.debug("Created state changelog topics {} from the parsed topology.", changelogTopicMetadata.values());
    // ---------------- Step Two ---------------- //
    // assign tasks to clients
    final Map<UUID, ClientState> states = new HashMap<>();
    for (final Map.Entry<UUID, ClientMetadata> entry : clientsMetadata.entrySet()) {
        states.put(entry.getKey(), entry.getValue().state);
    }
    log.debug("Assigning tasks {} to clients {} with number of replicas {}", partitionsForTask.keySet(), states, numStandbyReplicas);
    final StickyTaskAssignor<UUID> taskAssignor = new StickyTaskAssignor<>(states, partitionsForTask.keySet());
    taskAssignor.assign(numStandbyReplicas);
    log.info("Assigned tasks to clients as {}.", states);
    // ---------------- Step Three ---------------- //
    // construct the global partition assignment per host map
    final Map<HostInfo, Set<TopicPartition>> partitionsByHostState = new HashMap<>();
    if (minUserMetadataVersion == 2) {
        for (final Map.Entry<UUID, ClientMetadata> entry : clientsMetadata.entrySet()) {
            final HostInfo hostInfo = entry.getValue().hostInfo;
            if (hostInfo != null) {
                final Set<TopicPartition> topicPartitions = new HashSet<>();
                final ClientState state = entry.getValue().state;
                for (final TaskId id : state.activeTasks()) {
                    topicPartitions.addAll(partitionsForTask.get(id));
                }
                partitionsByHostState.put(hostInfo, topicPartitions);
            }
        }
    }
    taskManager.setPartitionsByHostState(partitionsByHostState);
    // within the client, distribute tasks to its owned consumers
    final Map<String, Assignment> assignment = new HashMap<>();
    for (final Map.Entry<UUID, ClientMetadata> entry : clientsMetadata.entrySet()) {
        final Set<String> consumers = entry.getValue().consumers;
        final ClientState state = entry.getValue().state;
        final List<List<TaskId>> interleavedActive = interleaveTasksByGroupId(state.activeTasks(), consumers.size());
        final List<List<TaskId>> interleavedStandby = interleaveTasksByGroupId(state.standbyTasks(), consumers.size());
        int consumerTaskIndex = 0;
        for (final String consumer : consumers) {
            final Map<TaskId, Set<TopicPartition>> standby = new HashMap<>();
            final ArrayList<AssignedPartition> assignedPartitions = new ArrayList<>();
            final List<TaskId> assignedActiveList = interleavedActive.get(consumerTaskIndex);
            for (final TaskId taskId : assignedActiveList) {
                for (final TopicPartition partition : partitionsForTask.get(taskId)) {
                    assignedPartitions.add(new AssignedPartition(taskId, partition));
                }
            }
            if (!state.standbyTasks().isEmpty()) {
                final List<TaskId> assignedStandbyList = interleavedStandby.get(consumerTaskIndex);
                for (final TaskId taskId : assignedStandbyList) {
                    Set<TopicPartition> standbyPartitions = standby.get(taskId);
                    if (standbyPartitions == null) {
                        standbyPartitions = new HashSet<>();
                        standby.put(taskId, standbyPartitions);
                    }
                    standbyPartitions.addAll(partitionsForTask.get(taskId));
                }
            }
            consumerTaskIndex++;
            Collections.sort(assignedPartitions);
            final List<TaskId> active = new ArrayList<>();
            final List<TopicPartition> activePartitions = new ArrayList<>();
            for (final AssignedPartition partition : assignedPartitions) {
                active.add(partition.taskId);
                activePartitions.add(partition.partition);
            }
            // finally, encode the assignment before sending back to coordinator
            assignment.put(consumer, new Assignment(activePartitions, new AssignmentInfo(minUserMetadataVersion, active, standby, partitionsByHostState).encode()));
        }
    }
    return assignment;
}

Example 34 with HostInfo

use of org.apache.kafka.streams.state.HostInfo in project apache-kafka-on-k8s by banzaicloud.

the class StreamsPartitionAssignor method onAssignment.

/**
 * @throws TaskAssignmentException if there is no task id for one of the partitions specified
 */
@Override
public void onAssignment(final Assignment assignment) {
    final List<TopicPartition> partitions = new ArrayList<>(assignment.partitions());
    Collections.sort(partitions, PARTITION_COMPARATOR);
    final AssignmentInfo info = AssignmentInfo.decode(assignment.userData());
    final int usedVersion = info.version();
    // version 1 field
    final Map<TaskId, Set<TopicPartition>> activeTasks = new HashMap<>();
    // version 2 fields
    final Map<TopicPartition, PartitionInfo> topicToPartitionInfo = new HashMap<>();
    final Map<HostInfo, Set<TopicPartition>> partitionsByHost;
    switch(usedVersion) {
        case 1:
            processVersionOneAssignment(info, partitions, activeTasks);
            partitionsByHost = Collections.emptyMap();
            break;
        case 2:
            processVersionTwoAssignment(info, partitions, activeTasks, topicToPartitionInfo);
            partitionsByHost = info.partitionsByHost();
            break;
        default:
            throw new IllegalStateException("Unknown metadata version: " + usedVersion + "; latest supported version: " + AssignmentInfo.LATEST_SUPPORTED_VERSION);
    }
    taskManager.setClusterMetadata(Cluster.empty().withPartitions(topicToPartitionInfo));
    taskManager.setPartitionsByHostState(partitionsByHost);
    taskManager.setAssignmentMetadata(activeTasks, info.standbyTasks());
    taskManager.updateSubscriptionsFromAssignment(partitions);
}

Example 35 with HostInfo

use of org.apache.kafka.streams.state.HostInfo in project apache-kafka-on-k8s by banzaicloud.

the class AssignmentInfo method encodePartitionsByHost.

private void encodePartitionsByHost(final DataOutputStream out) throws IOException {
    // encode partitions by host
    out.writeInt(partitionsByHost.size());
    for (final Map.Entry<HostInfo, Set<TopicPartition>> entry : partitionsByHost.entrySet()) {
        final HostInfo hostInfo = entry.getKey();
        out.writeUTF(hostInfo.host());
        out.writeInt(hostInfo.port());
        writeTopicPartitions(out, entry.getValue());
    }
}