Search in sources :

Example 1 with IncompatibleSchemaException

use of org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException in project pulsar by apache.

the class AbstractTopic method addSchema.

@Override
public CompletableFuture<SchemaVersion> addSchema(SchemaData schema) {
    if (schema == null) {
        return CompletableFuture.completedFuture(SchemaVersion.Empty);
    }
    String base = TopicName.get(getName()).getPartitionedTopicName();
    String id = TopicName.get(base).getSchemaName();
    SchemaRegistryService schemaRegistryService = brokerService.pulsar().getSchemaRegistryService();
    if (allowAutoUpdateSchema()) {
        return schemaRegistryService.putSchemaIfAbsent(id, schema, getSchemaCompatibilityStrategy());
    } else {
        return schemaRegistryService.trimDeletedSchemaAndGetList(id).thenCompose(schemaAndMetadataList -> schemaRegistryService.getSchemaVersionBySchemaData(schemaAndMetadataList, schema).thenCompose(schemaVersion -> {
            if (schemaVersion == null) {
                return FutureUtil.failedFuture(new IncompatibleSchemaException("Schema not found and schema auto updating is disabled."));
            } else {
                return CompletableFuture.completedFuture(schemaVersion);
            }
        }));
    }
}
Also used : StatsBuckets(org.apache.bookkeeper.mledger.util.StatsBuckets) Arrays(java.util.Arrays) BookkeeperSchemaStorage(org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage) LoggerFactory(org.slf4j.LoggerFactory) DelayedDeliveryPolicies(org.apache.pulsar.common.policies.data.DelayedDeliveryPolicies) ProducerBusyException(org.apache.pulsar.broker.service.BrokerServiceException.ProducerBusyException) SubscribeRate(org.apache.pulsar.common.policies.data.SubscribeRate) Preconditions.checkArgument(com.google.common.base.Preconditions.checkArgument) Pair(org.apache.commons.lang3.tuple.Pair) ResourceGroup(org.apache.pulsar.broker.resourcegroup.ResourceGroup) Map(java.util.Map) DispatchRateImpl(org.apache.pulsar.common.policies.data.impl.DispatchRateImpl) EnumSet(java.util.EnumSet) InactiveTopicPolicies(org.apache.pulsar.common.policies.data.InactiveTopicPolicies) SchemaCompatibilityStrategy(org.apache.pulsar.common.policies.data.SchemaCompatibilityStrategy) ResourceGroupPublishLimiter(org.apache.pulsar.broker.resourcegroup.ResourceGroupPublishLimiter) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) PublishRate(org.apache.pulsar.common.policies.data.PublishRate) HierarchyTopicPolicies(org.apache.pulsar.common.policies.data.HierarchyTopicPolicies) Objects(java.util.Objects) List(java.util.List) FutureUtil(org.apache.pulsar.common.util.FutureUtil) TopicTerminatedException(org.apache.pulsar.broker.service.BrokerServiceException.TopicTerminatedException) Optional(java.util.Optional) ENTRY_LATENCY_BUCKETS_USEC(org.apache.bookkeeper.mledger.impl.ManagedLedgerMBeanImpl.ENTRY_LATENCY_BUCKETS_USEC) Queue(java.util.Queue) IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException) ConcurrentLinkedQueue(java.util.concurrent.ConcurrentLinkedQueue) LongAdder(java.util.concurrent.atomic.LongAdder) ProducerFencedException(org.apache.pulsar.broker.service.BrokerServiceException.ProducerFencedException) TopicName(org.apache.pulsar.common.naming.TopicName) Getter(lombok.Getter) SchemaRegistryService(org.apache.pulsar.broker.service.schema.SchemaRegistryService) BacklogQuota(org.apache.pulsar.common.policies.data.BacklogQuota) CompletableFuture(java.util.concurrent.CompletableFuture) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) CollectionUtils(org.apache.commons.collections4.CollectionUtils) SubType(org.apache.pulsar.common.api.proto.CommandSubscribe.SubType) Lists(com.google.common.collect.Lists) MapUtils(org.apache.commons.collections4.MapUtils) RetentionPolicies(org.apache.pulsar.common.policies.data.RetentionPolicies) Summary(org.apache.pulsar.broker.stats.prometheus.metrics.Summary) Logger(org.slf4j.Logger) ServiceConfiguration(org.apache.pulsar.broker.ServiceConfiguration) MoreObjects(com.google.common.base.MoreObjects) AtomicLongFieldUpdater(java.util.concurrent.atomic.AtomicLongFieldUpdater) TopicPolicies(org.apache.pulsar.common.policies.data.TopicPolicies) SchemaVersion(org.apache.pulsar.common.protocol.schema.SchemaVersion) TimeUnit(java.util.concurrent.TimeUnit) Policies(org.apache.pulsar.common.policies.data.Policies) ConsumerBusyException(org.apache.pulsar.broker.service.BrokerServiceException.ConsumerBusyException) SchemaData(org.apache.pulsar.common.protocol.schema.SchemaData) Collections(java.util.Collections) IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException) SchemaRegistryService(org.apache.pulsar.broker.service.schema.SchemaRegistryService)

Example 2 with IncompatibleSchemaException

use of org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException in project pulsar by apache.

the class AvroSchemaBasedCompatibilityCheck method checkCompatible.

@Override
public void checkCompatible(Iterable<SchemaData> from, SchemaData to, SchemaCompatibilityStrategy strategy) throws IncompatibleSchemaException {
    LinkedList<Schema> fromList = new LinkedList<>();
    checkArgument(from != null, "check compatibility list is null");
    try {
        for (SchemaData schemaData : from) {
            Schema.Parser parser = new Schema.Parser();
            parser.setValidateDefaults(false);
            fromList.addFirst(parser.parse(new String(schemaData.getData(), UTF_8)));
        }
        Schema.Parser parser = new Schema.Parser();
        parser.setValidateDefaults(false);
        Schema toSchema = parser.parse(new String(to.getData(), UTF_8));
        SchemaValidator schemaValidator = createSchemaValidator(strategy);
        schemaValidator.validate(toSchema, fromList);
    } catch (SchemaParseException e) {
        log.warn("Error during schema parsing: {}", e.getMessage());
        throw new IncompatibleSchemaException(e);
    } catch (SchemaValidationException e) {
        log.warn("Error during schema compatibility check: {}", e.getMessage());
        throw new IncompatibleSchemaException(e);
    }
}
Also used : IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException) SchemaValidationException(org.apache.avro.SchemaValidationException) SchemaData(org.apache.pulsar.common.protocol.schema.SchemaData) SchemaParseException(org.apache.avro.SchemaParseException) Schema(org.apache.avro.Schema) SchemaValidator(org.apache.avro.SchemaValidator) LinkedList(java.util.LinkedList)

Example 3 with IncompatibleSchemaException

use of org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException in project pulsar by apache.

the class SchemaRegistryServiceImpl method checkCompatibilityWithAll.

private CompletableFuture<Void> checkCompatibilityWithAll(SchemaData schema, SchemaCompatibilityStrategy strategy, List<SchemaAndMetadata> schemaAndMetadataList) {
    CompletableFuture<Void> result = new CompletableFuture<>();
    if (strategy == SchemaCompatibilityStrategy.ALWAYS_COMPATIBLE) {
        result.complete(null);
    } else {
        SchemaAndMetadata breakSchema = null;
        for (SchemaAndMetadata schemaAndMetadata : schemaAndMetadataList) {
            if (schemaAndMetadata.schema.getType() != schema.getType()) {
                breakSchema = schemaAndMetadata;
                break;
            }
        }
        if (breakSchema == null) {
            try {
                compatibilityChecks.getOrDefault(schema.getType(), SchemaCompatibilityCheck.DEFAULT).checkCompatible(schemaAndMetadataList.stream().map(schemaAndMetadata -> schemaAndMetadata.schema).collect(Collectors.toList()), schema, strategy);
                result.complete(null);
            } catch (Exception e) {
                if (e instanceof IncompatibleSchemaException) {
                    result.completeExceptionally(e);
                } else {
                    result.completeExceptionally(new IncompatibleSchemaException(e));
                }
            }
        } else {
            result.completeExceptionally(new IncompatibleSchemaException(String.format("Incompatible schema: exists schema type %s, new schema type %s", breakSchema.schema.getType(), schema.getType())));
        }
    }
    return result;
}
Also used : IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException) CompletableFuture(java.util.concurrent.CompletableFuture) SchemaException(org.apache.pulsar.broker.service.schema.exceptions.SchemaException) InvalidProtocolBufferException(com.google.protobuf.InvalidProtocolBufferException) IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException)

Example 4 with IncompatibleSchemaException

use of org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException in project pulsar by apache.

the class JsonSchemaCompatibilityCheck method isCompatibleJsonSchema.

private void isCompatibleJsonSchema(SchemaData from, SchemaData to) throws IncompatibleSchemaException {
    try {
        ObjectMapper objectMapper = getObjectMapper();
        JsonSchema fromSchema = objectMapper.readValue(from.getData(), JsonSchema.class);
        JsonSchema toSchema = objectMapper.readValue(to.getData(), JsonSchema.class);
        if (!fromSchema.getId().equals(toSchema.getId())) {
            throw new IncompatibleSchemaException(String.format("Incompatible Schema from %s + to %s", new String(from.getData(), UTF_8), new String(to.getData(), UTF_8)));
        }
    } catch (IOException e) {
        throw new IncompatibleSchemaException(e);
    }
}
Also used : IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException) JsonSchema(com.fasterxml.jackson.module.jsonSchema.JsonSchema) IOException(java.io.IOException) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper)

Example 5 with IncompatibleSchemaException

use of org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException in project pulsar by apache.

the class ProtobufNativeSchemaCompatibilityCheck method checkCompatible.

@Override
public void checkCompatible(SchemaData from, SchemaData to, SchemaCompatibilityStrategy strategy) throws IncompatibleSchemaException {
    Descriptor fromDescriptor = ProtobufNativeSchemaUtils.deserialize(from.getData());
    Descriptor toDescriptor = ProtobufNativeSchemaUtils.deserialize(to.getData());
    switch(strategy) {
        case BACKWARD_TRANSITIVE:
        case BACKWARD:
        case FORWARD_TRANSITIVE:
        case FORWARD:
        case FULL_TRANSITIVE:
        case FULL:
            checkRootMessageChange(fromDescriptor, toDescriptor, strategy);
            return;
        case ALWAYS_COMPATIBLE:
            return;
        default:
            throw new IncompatibleSchemaException("Unknown SchemaCompatibilityStrategy.");
    }
}
Also used : IncompatibleSchemaException(org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException) Descriptor(com.google.protobuf.Descriptors.Descriptor)

Aggregations

IncompatibleSchemaException (org.apache.pulsar.broker.service.schema.exceptions.IncompatibleSchemaException)30 SchemaData (org.apache.pulsar.common.protocol.schema.SchemaData)15 CompletableFuture (java.util.concurrent.CompletableFuture)13 SchemaVersion (org.apache.pulsar.common.protocol.schema.SchemaVersion)9 IOException (java.io.IOException)6 LinkedList (java.util.LinkedList)6 List (java.util.List)6 TopicName (org.apache.pulsar.common.naming.TopicName)6 Logger (org.slf4j.Logger)6 LoggerFactory (org.slf4j.LoggerFactory)6 Pair (org.apache.commons.lang3.tuple.Pair)4 SchemaType (org.apache.pulsar.common.schema.SchemaType)4 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)3 JsonSchema (com.fasterxml.jackson.module.jsonSchema.JsonSchema)3 InvalidProtocolBufferException (com.google.protobuf.InvalidProtocolBufferException)3 SchemaException (org.apache.pulsar.broker.service.schema.exceptions.SchemaException)3 SchemaHash (org.apache.pulsar.common.protocol.schema.SchemaHash)3 FutureUtil (org.apache.pulsar.common.util.FutureUtil)3 VisibleForTesting (com.google.common.annotations.VisibleForTesting)2 Charsets (com.google.common.base.Charsets)2