Search in sources :

Example 1 with ManagedConnector

use of org.bf2.cos.fleetshard.api.ManagedConnector in project cos-fleetshard by bf2fc6cc711aee1a0c2a.

the class ConnectorDeploymentProvisioner method provision.

public void provision(ConnectorDeployment deployment) {
    final String uow = uid();
    LOGGER.info("Got cluster_id: {}, connector_id: {}, deployment_id: {}, resource_version: {}, uow: {}", fleetShard.getClusterId(), deployment.getSpec().getConnectorId(), deployment.getId(), deployment.getMetadata().getResourceVersion(), uow);
    // TODO: cache cluster
    final ManagedConnectorCluster cluster = fleetShard.getOrCreateManagedConnectorCluster();
    final ManagedConnector connector = createManagedConnector(uow, deployment, cluster);
    final Secret secret = createManagedConnectorSecret(uow, deployment, connector);
    LOGGER.info("CreateOrReplace - uow: {}, managed_connector: {}/{}, managed_connector_secret: {}/{}", uow, connector.getMetadata().getNamespace(), connector.getMetadata().getName(), secret.getMetadata().getNamespace(), secret.getMetadata().getName());
}
Also used : Secret(io.fabric8.kubernetes.api.model.Secret) ManagedConnectorCluster(org.bf2.cos.fleetshard.api.ManagedConnectorCluster) ManagedConnector(org.bf2.cos.fleetshard.api.ManagedConnector)

Example 2 with ManagedConnector

use of org.bf2.cos.fleetshard.api.ManagedConnector in project cos-fleetshard by bf2fc6cc711aee1a0c2a.

the class CamelOperandSupport method createSteps.

public static List<ProcessorKamelet> createSteps(ManagedConnector connector, ConnectorConfiguration<ObjectNode> connectorConfiguration, CamelShardMetadata shardMetadata, Map<String, String> props) {
    String consumes = Optional.ofNullable(connectorConfiguration.getDataShapeSpec()).map(spec -> spec.at("/consumes/format")).filter(node -> !node.isMissingNode()).map(JsonNode::asText).orElse(shardMetadata.getConsumes());
    String produces = Optional.ofNullable(connectorConfiguration.getDataShapeSpec()).map(spec -> spec.at("/produces/format")).filter(node -> !node.isMissingNode()).map(JsonNode::asText).orElse(shardMetadata.getProduces());
    final ArrayNode steps = connectorConfiguration.getProcessorsSpec();
    final List<ProcessorKamelet> stepDefinitions = new ArrayList<>(steps.size() + 2);
    int i = 0;
    if (consumes != null) {
        switch(consumes) {
            case "application/json":
                {
                    String stepName = stepName(i, "cos-decoder-json-action");
                    stepDefinitions.add(new ProcessorKamelet("cos-decoder-json-action", stepName));
                    if (shardMetadata.getConsumesClass() != null) {
                        props.put(kameletProperty("cos-decoder-json-action", stepName, "contentClass"), shardMetadata.getConsumesClass());
                    }
                    i++;
                }
                break;
            case "avro/binary":
                {
                    String stepName = stepName(i, "cos-decoder-avro-action");
                    stepDefinitions.add(new ProcessorKamelet("cos-decoder-avro-action", stepName));
                    if (shardMetadata.getConsumesClass() != null) {
                        props.put(kameletProperty("cos-decoder-avro-action", stepName, "contentClass"), shardMetadata.getConsumesClass());
                    }
                    i++;
                }
                break;
            case "application/x-java-object":
                {
                    String stepName = stepName(i, "cos-decoder-pojo-action");
                    stepDefinitions.add(new ProcessorKamelet("cos-decoder-pojo-action", stepName));
                    if (produces != null) {
                        props.put(kameletProperty("cos-decoder-pojo-action", stepName, "mimeType"), produces);
                    }
                    i++;
                }
                break;
            case "text/plain":
            case "application/octet-stream":
                break;
            default:
                throw new IllegalArgumentException("Unsupported value format " + consumes);
        }
    }
    for (JsonNode step : steps) {
        var element = step.fields().next();
        String templateId = shardMetadata.getKamelets().getProcessors().get(element.getKey());
        if (templateId == null) {
            throw new IllegalArgumentException("Unknown processor: " + element.getKey());
        }
        stepDefinitions.add(new ProcessorKamelet(templateId, stepName(i, templateId)));
        configureStep(props, (ObjectNode) element.getValue(), i, shardMetadata.getKamelets().getProcessors().get(element.getKey()));
        i++;
    }
    if (produces != null) {
        switch(produces) {
            case "application/json":
                {
                    String stepName = stepName(i, "cos-encoder-json-action");
                    stepDefinitions.add(new ProcessorKamelet("cos-encoder-json-action", stepName));
                    if (shardMetadata.getProducesClass() != null) {
                        props.put(kameletProperty("cos-encoder-json-action", stepName, "contentClass"), shardMetadata.getProducesClass());
                    }
                }
                break;
            case "avro/binary":
                {
                    String stepName = stepName(i, "cos-encoder-avro-action");
                    stepDefinitions.add(new ProcessorKamelet("cos-encoder-avro-action", stepName));
                    if (shardMetadata.getProducesClass() != null) {
                        props.put(kameletProperty("cos-encoder-avro-action", stepName, "contentClass"), shardMetadata.getProducesClass());
                    }
                }
                break;
            case "text/plain":
                {
                    stepDefinitions.add(new ProcessorKamelet("cos-encoder-string-action", stepName(i, "cos-encoder-string-action")));
                }
                break;
            case "application/octet-stream":
                {
                    stepDefinitions.add(new ProcessorKamelet("cos-encoder-bytearray-action", stepName(i, "cos-encoder-bytearray-action")));
                }
                break;
            default:
                throw new IllegalArgumentException("Unsupported value format " + produces);
        }
    }
    // If it is a sink, then it consumes from kafka
    if (isSink(shardMetadata)) {
        props.put(String.format("camel.kamelet.%s.valueDeserializer", shardMetadata.getKamelets().getKafka().getName()), "org.bf2.cos.connector.camel.serdes.bytes.ByteArrayDeserializer");
        if ("application/json".equals(consumes) && hasSchemaRegistry(connector)) {
            props.put(String.format("camel.kamelet.%s.valueDeserializer", shardMetadata.getKamelets().getKafka().getName()), "org.bf2.cos.connector.camel.serdes.json.JsonDeserializer");
        }
        if ("avro/binary".equals(produces) && hasSchemaRegistry(connector)) {
            props.put(String.format("camel.kamelet.%s.valueDeserializer", shardMetadata.getKamelets().getKafka().getName()), "org.bf2.cos.connector.camel.serdes.avro.AvroDeserializer");
        }
    }
    // If it is a source, then it produces to kafka
    if (isSource(shardMetadata)) {
        props.put(String.format("camel.kamelet.%s.valueSerializer", shardMetadata.getKamelets().getKafka().getName()), "org.bf2.cos.connector.camel.serdes.bytes.ByteArraySerializer");
        if ("application/json".equals(produces) && hasSchemaRegistry(connector)) {
            props.put(String.format("camel.kamelet.%s.valueSerializer", shardMetadata.getKamelets().getKafka().getName()), "org.bf2.cos.connector.camel.serdes.json.JsonSerializer");
        }
        if ("avro/binary".equals(produces) && hasSchemaRegistry(connector)) {
            props.put(String.format("camel.kamelet.%s.valueSerializer", shardMetadata.getKamelets().getKafka().getName()), "org.bf2.cos.connector.camel.serdes.avro.AvroSerializer");
        }
    }
    return stepDefinitions;
}
Also used : ConnectorStatusSpec(org.bf2.cos.fleetshard.api.ConnectorStatusSpec) ERROR_HANDLER_LOG_TYPE(org.bf2.cos.fleetshard.operator.camel.CamelConstants.ERROR_HANDLER_LOG_TYPE) StringUtils(org.apache.commons.lang3.StringUtils) KameletBindingStatus(org.bf2.cos.fleetshard.operator.camel.model.KameletBindingStatus) ObjectNode(com.fasterxml.jackson.databind.node.ObjectNode) ArrayList(java.util.ArrayList) ERROR_HANDLER_STOP_URI(org.bf2.cos.fleetshard.operator.camel.CamelConstants.ERROR_HANDLER_STOP_URI) Serialization(io.fabric8.kubernetes.client.utils.Serialization) Locale(java.util.Locale) Map(java.util.Map) JsonNode(com.fasterxml.jackson.databind.JsonNode) ERROR_HANDLER_SINK_CHANNEL_TYPE(org.bf2.cos.fleetshard.operator.camel.CamelConstants.ERROR_HANDLER_SINK_CHANNEL_TYPE) ManagedConnector(org.bf2.cos.fleetshard.api.ManagedConnector) ConnectorConfiguration(org.bf2.cos.fleetshard.operator.connector.ConnectorConfiguration) ERROR_HANDLER_DEAD_LETTER_CHANNEL_KAMELET(org.bf2.cos.fleetshard.operator.camel.CamelConstants.ERROR_HANDLER_DEAD_LETTER_CHANNEL_KAMELET) CamelShardMetadata(org.bf2.cos.fleetshard.operator.camel.model.CamelShardMetadata) Iterator(java.util.Iterator) ProcessorKamelet(org.bf2.cos.fleetshard.operator.camel.model.ProcessorKamelet) CaseUtils(org.apache.commons.text.CaseUtils) ServiceAccountSpec(org.bf2.cos.fleetshard.api.ServiceAccountSpec) KameletBinding(org.bf2.cos.fleetshard.operator.camel.model.KameletBinding) StandardCharsets(java.nio.charset.StandardCharsets) ArrayNode(com.fasterxml.jackson.databind.node.ArrayNode) Base64(java.util.Base64) List(java.util.List) Condition(io.fabric8.kubernetes.api.model.Condition) CONNECTOR_TYPE_SOURCE(org.bf2.cos.fleetshard.operator.camel.CamelConstants.CONNECTOR_TYPE_SOURCE) ERROR_HANDLER_DEAD_LETTER_CHANNEL_KAMELET_ID(org.bf2.cos.fleetshard.operator.camel.CamelConstants.ERROR_HANDLER_DEAD_LETTER_CHANNEL_KAMELET_ID) KubernetesClient(io.fabric8.kubernetes.client.KubernetesClient) Optional(java.util.Optional) CONNECTOR_TYPE_SINK(org.bf2.cos.fleetshard.operator.camel.CamelConstants.CONNECTOR_TYPE_SINK) Kamelets(org.bf2.cos.fleetshard.operator.camel.model.Kamelets) JacksonUtil.iterator(org.bf2.cos.fleetshard.support.json.JacksonUtil.iterator) ProcessorKamelet(org.bf2.cos.fleetshard.operator.camel.model.ProcessorKamelet) ArrayList(java.util.ArrayList) JsonNode(com.fasterxml.jackson.databind.JsonNode) ArrayNode(com.fasterxml.jackson.databind.node.ArrayNode)

Example 3 with ManagedConnector

use of org.bf2.cos.fleetshard.api.ManagedConnector in project cos-fleetshard by bf2fc6cc711aee1a0c2a.

the class ConnectorDeploymentProvisioner method provision.

public void provision(ConnectorDeployment deployment) {
    final String uow = uid();
    LOGGER.info("Got cluster_id: {}, namespace_d: {}, connector_id: {}, deployment_id: {}, resource_version: {}, uow: {}", fleetShard.getClusterId(), deployment.getSpec().getNamespaceId(), deployment.getSpec().getConnectorId(), deployment.getId(), deployment.getMetadata().getResourceVersion(), uow);
    final ManagedConnector connector = createManagedConnector(uow, deployment, null);
    final Secret secret = createManagedConnectorSecret(uow, deployment, connector);
    LOGGER.info("CreateOrReplace - uow: {}, connector: {}/{}, secret: {}/{}", uow, connector.getMetadata().getNamespace(), connector.getMetadata().getName(), secret.getMetadata().getNamespace(), secret.getMetadata().getName());
}
Also used : Secret(io.fabric8.kubernetes.api.model.Secret) ManagedConnector(org.bf2.cos.fleetshard.api.ManagedConnector)

Example 4 with ManagedConnector

use of org.bf2.cos.fleetshard.api.ManagedConnector in project cos-fleetshard by bf2fc6cc711aee1a0c2a.

the class ConnectorStatusExtractor method extract.

public static ConnectorDeploymentStatus extract(ManagedConnector connector) {
    ConnectorDeploymentStatus status = new ConnectorDeploymentStatus();
    DeploymentSpec deployment = connector.getSpec().getDeployment();
    if (connector.getStatus() != null && connector.getStatus().getPhase() != null) {
        deployment = connector.getStatus().getDeployment();
    }
    status.setResourceVersion(deployment.getDeploymentResourceVersion());
    if (connector.getSpec().getOperatorSelector() == null || connector.getSpec().getOperatorSelector().getId() == null) {
        status.setPhase(ConnectorState.FAILED);
        status.addConditionsItem(new MetaV1Condition().type(Conditions.TYPE_READY).status(Conditions.STATUS_FALSE).message("No assignable operator").reason(Conditions.NO_ASSIGNABLE_OPERATOR_REASON).lastTransitionTime(Conditions.now()));
        return status;
    }
    if (connector.getStatus() != null && connector.getStatus().getConnectorStatus() != null) {
        status.setOperators(new ConnectorDeploymentStatusOperators().assigned(toConnectorOperator(connector.getStatus().getConnectorStatus().getAssignedOperator())).available(toConnectorOperator(connector.getStatus().getConnectorStatus().getAvailableOperator())));
        if (connector.getStatus().getConnectorStatus() != null) {
            if (connector.getStatus().getConnectorStatus().getPhase() != null) {
                status.setPhase(ConnectorState.fromValue(connector.getStatus().getConnectorStatus().getPhase()));
            }
            if (connector.getStatus().getConnectorStatus().getConditions() != null) {
                for (var cond : connector.getStatus().getConnectorStatus().getConditions()) {
                    status.addConditionsItem(toMetaV1Condition(cond));
                }
            }
        }
    }
    if (status.getPhase() == null) {
        status.setPhase(ConnectorState.PROVISIONING);
        if (DESIRED_STATE_DELETED.equals(deployment.getDesiredState())) {
            status.setPhase(ConnectorState.DEPROVISIONING);
        } else if (DESIRED_STATE_STOPPED.equals(deployment.getDesiredState())) {
            status.setPhase(ConnectorState.DEPROVISIONING);
        } else if (DESIRED_STATE_UNASSIGNED.equals(deployment.getDesiredState())) {
            status.setPhase(ConnectorState.DEPROVISIONING);
        }
    }
    return status;
}
Also used : DeploymentSpec(org.bf2.cos.fleetshard.api.DeploymentSpec) ConnectorDeploymentStatusOperators(org.bf2.cos.fleet.manager.model.ConnectorDeploymentStatusOperators) MetaV1Condition(org.bf2.cos.fleet.manager.model.MetaV1Condition) ConnectorDeploymentStatus(org.bf2.cos.fleet.manager.model.ConnectorDeploymentStatus)

Example 5 with ManagedConnector

use of org.bf2.cos.fleetshard.api.ManagedConnector in project cos-fleetshard by bf2fc6cc711aee1a0c2a.

the class ConnectorStatusSync method sync.

private void sync() {
    int count = 0;
    try {
        for (ManagedConnector connector : connectorClient.getAllConnectors()) {
            updater.update(connector);
            count++;
        }
    } finally {
        if (count > 0) {
            Counter.builder(config.metrics().baseName() + "." + METRICS_SYNC + ".total").register(registry).increment(count);
        }
    }
}
Also used : ManagedConnector(org.bf2.cos.fleetshard.api.ManagedConnector)

Aggregations

ManagedConnector (org.bf2.cos.fleetshard.api.ManagedConnector)24 Secret (io.fabric8.kubernetes.api.model.Secret)15 ObjectMetaBuilder (io.fabric8.kubernetes.api.model.ObjectMetaBuilder)9 ObjectNode (com.fasterxml.jackson.databind.node.ObjectNode)8 Map (java.util.Map)6 FleetShardSyncConfig (org.bf2.cos.fleetshard.sync.FleetShardSyncConfig)6 FleetManagerClient (org.bf2.cos.fleetshard.sync.client.FleetManagerClient)6 FleetShardClient (org.bf2.cos.fleetshard.sync.client.FleetShardClient)6 Test (org.junit.jupiter.api.Test)6 SecretBuilder (io.fabric8.kubernetes.api.model.SecretBuilder)5 MeterRegistry (io.micrometer.core.instrument.MeterRegistry)5 List (java.util.List)5 ConnectorDeployment (org.bf2.cos.fleet.manager.model.ConnectorDeployment)5 Operator (org.bf2.cos.fleetshard.api.Operator)5 ServiceAccountSpec (org.bf2.cos.fleetshard.api.ServiceAccountSpec)5 ArrayNode (com.fasterxml.jackson.databind.node.ArrayNode)4 HasMetadata (io.fabric8.kubernetes.api.model.HasMetadata)4 KubernetesClient (io.fabric8.kubernetes.client.KubernetesClient)4 ManagedConnectorBuilder (org.bf2.cos.fleetshard.api.ManagedConnectorBuilder)4 ConnectorConfiguration (org.bf2.cos.fleetshard.operator.connector.ConnectorConfiguration)4