Search in sources :

Example 11 with SchemaMetadata

use of com.hortonworks.registries.schemaregistry.SchemaMetadata in project registry by hortonworks.

the class KafkaAvroSerializer method createSchemaMetadata.

private SchemaMetadata createSchemaMetadata(String topic) {
    SchemaMetadata schemaMetadata = getSchemaKey(topic, isKey);
    String description = "Schema registered by KafkaAvroSerializer for topic: [" + topic + "] iskey: [" + isKey + "]";
    return new SchemaMetadata.Builder(schemaMetadata).description(description).compatibility(compatibility).build();
}
Also used : SchemaMetadata(com.hortonworks.registries.schemaregistry.SchemaMetadata)

Example 12 with SchemaMetadata

use of com.hortonworks.registries.schemaregistry.SchemaMetadata in project registry by hortonworks.

the class ConfluentSchemaRegistryCompatibleResource method registerSchemaVersion.

@POST
@Path("/subjects/{subject}/versions")
@ApiOperation(value = "Register a new version of the schema", notes = "Registers the given schema version to schema with subject if the given schemaText is not registered as a version for this schema, " + "and returns respective unique id." + "In case of incompatible schema errors, it throws error message like 'Unable to read schema: <> using schema <>' ", response = Id.class, tags = OPERATION_GROUP_CONFLUENT_SR)
@Timed
@UnitOfWork
public Response registerSchemaVersion(@ApiParam(value = "subject", required = true) @PathParam("subject") String subject, @ApiParam(value = "Details about the schema", required = true) String schema, @Context UriInfo uriInfo) {
    LOG.info("registerSchema for [{}] is [{}]", subject);
    return handleLeaderAction(uriInfo, () -> {
        Response response;
        try {
            LOG.info("registerSchema for [{}] is [{}]", subject);
            SchemaMetadataInfo schemaMetadataInfo = schemaRegistry.getSchemaMetadataInfo(subject);
            if (schemaMetadataInfo == null) {
                SchemaMetadata schemaMetadata = new SchemaMetadata.Builder(subject).type(AvroSchemaProvider.TYPE).schemaGroup("Kafka").build();
                schemaRegistry.addSchemaMetadata(schemaMetadata);
                schemaMetadataInfo = schemaRegistry.getSchemaMetadataInfo(subject);
            }
            SchemaIdVersion schemaVersionInfo = schemaRegistry.addSchemaVersion(schemaMetadataInfo.getSchemaMetadata(), new SchemaVersion(schemaStringFromJson(schema).getSchema(), null));
            Id id = new Id();
            id.setId(schemaVersionInfo.getSchemaVersionId());
            response = WSUtils.respondEntity(id, Response.Status.OK);
        } catch (InvalidSchemaException ex) {
            LOG.error("Invalid schema error encountered while adding subject [{}]", subject, ex);
            response = invalidSchemaError();
        } catch (IncompatibleSchemaException ex) {
            LOG.error("Incompatible schema error encountered while adding subject [{}]", subject, ex);
            response = incompatibleSchemaError();
        } catch (UnsupportedSchemaTypeException ex) {
            LOG.error("Unsupported schema type encountered while adding subject [{}]", subject, ex);
            response = incompatibleSchemaError();
        } catch (Exception ex) {
            LOG.error("Encountered error while adding subject [{}]", subject, ex);
            response = serverError();
        }
        return response;
    });
}
Also used : CatalogResponse(com.hortonworks.registries.common.catalog.CatalogResponse) Response(javax.ws.rs.core.Response) IncompatibleSchemaException(com.hortonworks.registries.schemaregistry.errors.IncompatibleSchemaException) InvalidSchemaException(com.hortonworks.registries.schemaregistry.errors.InvalidSchemaException) SchemaMetadata(com.hortonworks.registries.schemaregistry.SchemaMetadata) SchemaVersion(com.hortonworks.registries.schemaregistry.SchemaVersion) UnsupportedSchemaTypeException(com.hortonworks.registries.schemaregistry.errors.UnsupportedSchemaTypeException) SchemaIdVersion(com.hortonworks.registries.schemaregistry.SchemaIdVersion) UnsupportedSchemaTypeException(com.hortonworks.registries.schemaregistry.errors.UnsupportedSchemaTypeException) IncompatibleSchemaException(com.hortonworks.registries.schemaregistry.errors.IncompatibleSchemaException) InvalidSchemaException(com.hortonworks.registries.schemaregistry.errors.InvalidSchemaException) IOException(java.io.IOException) SchemaNotFoundException(com.hortonworks.registries.schemaregistry.errors.SchemaNotFoundException) SchemaMetadataInfo(com.hortonworks.registries.schemaregistry.SchemaMetadataInfo) Path(javax.ws.rs.Path) UnitOfWork(com.hortonworks.registries.common.transaction.UnitOfWork) POST(javax.ws.rs.POST) Timed(com.codahale.metrics.annotation.Timed) ApiOperation(io.swagger.annotations.ApiOperation)

Example 13 with SchemaMetadata

use of com.hortonworks.registries.schemaregistry.SchemaMetadata in project registry by hortonworks.

the class SchemaRegistryResource method findAggregatedSchemas.

@GET
@Path("/search/schemas/aggregated")
@ApiOperation(value = "Search for schemas containing the given name and description", notes = "Search the schemas for given name and description, return a list of schemas that contain the field.", response = AggregatedSchemaMetadataInfo.class, responseContainer = "Collection", tags = OPERATION_GROUP_SCHEMA)
@Timed
@UnitOfWork
public Response findAggregatedSchemas(@Context UriInfo uriInfo) {
    MultivaluedMap<String, String> queryParameters = uriInfo.getQueryParameters();
    try {
        Collection<SchemaMetadataInfo> schemaMetadataInfos = findSchemaMetadataInfos(uriInfo.getQueryParameters());
        List<AggregatedSchemaMetadataInfo> aggregatedSchemaMetadataInfos = new ArrayList<>();
        for (SchemaMetadataInfo schemaMetadataInfo : schemaMetadataInfos) {
            SchemaMetadata schemaMetadata = schemaMetadataInfo.getSchemaMetadata();
            List<SerDesInfo> serDesInfos = new ArrayList<>(schemaRegistry.getSerDes(schemaMetadataInfo.getSchemaMetadata().getName()));
            aggregatedSchemaMetadataInfos.add(new AggregatedSchemaMetadataInfo(schemaMetadata, schemaMetadataInfo.getId(), schemaMetadataInfo.getTimestamp(), schemaRegistry.getAggregatedSchemaBranch(schemaMetadata.getName()), serDesInfos));
        }
        return WSUtils.respondEntities(aggregatedSchemaMetadataInfos, Response.Status.OK);
    } catch (SchemaBranchNotFoundException e) {
        return WSUtils.respond(Response.Status.NOT_FOUND, CatalogResponse.ResponseMessage.ENTITY_NOT_FOUND, e.getMessage());
    } catch (Exception ex) {
        LOG.error("Encountered error while finding schemas for given fields [{}]", queryParameters, ex);
        return WSUtils.respond(Response.Status.INTERNAL_SERVER_ERROR, CatalogResponse.ResponseMessage.EXCEPTION, ex.getMessage());
    }
}
Also used : SchemaMetadata(com.hortonworks.registries.schemaregistry.SchemaMetadata) SchemaBranchNotFoundException(com.hortonworks.registries.schemaregistry.errors.SchemaBranchNotFoundException) ArrayList(java.util.ArrayList) AggregatedSchemaMetadataInfo(com.hortonworks.registries.schemaregistry.AggregatedSchemaMetadataInfo) SerDesInfo(com.hortonworks.registries.schemaregistry.SerDesInfo) SchemaBranchNotFoundException(com.hortonworks.registries.schemaregistry.errors.SchemaBranchNotFoundException) InvalidSchemaBranchDeletionException(com.hortonworks.registries.schemaregistry.errors.InvalidSchemaBranchDeletionException) FileNotFoundException(java.io.FileNotFoundException) SchemaNotFoundException(com.hortonworks.registries.schemaregistry.errors.SchemaNotFoundException) SchemaBranchAlreadyExistsException(com.hortonworks.registries.schemaregistry.errors.SchemaBranchAlreadyExistsException) UnsupportedSchemaTypeException(com.hortonworks.registries.schemaregistry.errors.UnsupportedSchemaTypeException) IncompatibleSchemaException(com.hortonworks.registries.schemaregistry.errors.IncompatibleSchemaException) InvalidSchemaException(com.hortonworks.registries.schemaregistry.errors.InvalidSchemaException) SchemaLifecycleException(com.hortonworks.registries.schemaregistry.state.SchemaLifecycleException) IOException(java.io.IOException) AggregatedSchemaMetadataInfo(com.hortonworks.registries.schemaregistry.AggregatedSchemaMetadataInfo) SchemaMetadataInfo(com.hortonworks.registries.schemaregistry.SchemaMetadataInfo) Path(javax.ws.rs.Path) UnitOfWork(com.hortonworks.registries.common.transaction.UnitOfWork) Timed(com.codahale.metrics.annotation.Timed) GET(javax.ws.rs.GET) ApiOperation(io.swagger.annotations.ApiOperation)

Example 14 with SchemaMetadata

use of com.hortonworks.registries.schemaregistry.SchemaMetadata in project registry by hortonworks.

the class BasicSchemaRegistryClientOpsTest method doTestSchemaOps.

private void doTestSchemaOps(SchemaValidationLevel validationLevel) throws IOException, InvalidSchemaException, IncompatibleSchemaException, SchemaNotFoundException, SchemaBranchNotFoundException {
    String testName = TEST_NAME_RULE.getMethodName();
    SchemaMetadata schemaMetadata = new SchemaMetadata.Builder(testName + "-schema").type(AvroSchemaProvider.TYPE).schemaGroup(testName + "-group").description("Schema for " + testName).validationLevel(validationLevel).compatibility(SchemaCompatibility.BOTH).build();
    Long id = schemaRegistryClient.registerSchemaMetadata(schemaMetadata);
    Assert.assertNotNull(id);
    // registering a new schema
    String schemaName = schemaMetadata.getName();
    String schema1 = AvroSchemaRegistryClientUtil.getSchema("/schema-1.avsc");
    SchemaIdVersion v1 = schemaRegistryClient.addSchemaVersion(schemaName, new SchemaVersion(schema1, "Initial version of the schema"));
    Assert.assertNotNull(v1.getSchemaMetadataId());
    Assert.assertEquals(1, v1.getVersion().intValue());
    SchemaMetadataInfo schemaMetadataInfoForId = schemaRegistryClient.getSchemaMetadataInfo(v1.getSchemaMetadataId());
    SchemaMetadataInfo schemaMetadataInfoForName = schemaRegistryClient.getSchemaMetadataInfo(schemaName);
    Assert.assertEquals(schemaMetadataInfoForId, schemaMetadataInfoForName);
    // adding a new version of the schema using uploadSchemaVersion API
    SchemaIdVersion v2 = schemaRegistryClient.uploadSchemaVersion(schemaMetadata.getName(), "second version", AvroSchemaRegistryClientTest.class.getResourceAsStream("/schema-2.avsc"));
    Assert.assertEquals(v1.getVersion() + 1, v2.getVersion().intValue());
    SchemaVersionInfo schemaVersionInfo = schemaRegistryClient.getSchemaVersionInfo(new SchemaVersionKey(schemaName, v2.getVersion()));
    SchemaVersionInfo latest = schemaRegistryClient.getLatestSchemaVersionInfo(schemaName);
    Assert.assertEquals(latest, schemaVersionInfo);
    Collection<SchemaVersionInfo> allVersions = schemaRegistryClient.getAllVersions(schemaName);
    Assert.assertEquals(2, allVersions.size());
    // receive the same version as earlier without adding a new schema entry as it exists in the same schema group.
    SchemaIdVersion version = schemaRegistryClient.addSchemaVersion(schemaMetadata, new SchemaVersion(schema1, "already added schema"));
    Assert.assertEquals(version, v1);
    Collection<SchemaVersionKey> md5SchemaVersionKeys = schemaRegistryClient.findSchemasByFields(new SchemaFieldQuery.Builder().name("md5").build());
    Assert.assertEquals(2, md5SchemaVersionKeys.size());
    Collection<SchemaVersionKey> txidSchemaVersionKeys = schemaRegistryClient.findSchemasByFields(new SchemaFieldQuery.Builder().name("txid").build());
    Assert.assertEquals(1, txidSchemaVersionKeys.size());
    // checks we can update schema meta data.
    SchemaMetadata currentSchemaMetadata = schemaRegistryClient.getSchemaMetadataInfo(schemaName).getSchemaMetadata();
    SchemaMetadata schemaMetadataToUpdateTo = new SchemaMetadata.Builder(currentSchemaMetadata).validationLevel(SchemaValidationLevel.LATEST).build();
    SchemaMetadataInfo updatedSchemaMetadata = schemaRegistryClient.updateSchemaMetadata(schemaName, schemaMetadataToUpdateTo);
    Assert.assertEquals(SchemaValidationLevel.LATEST, updatedSchemaMetadata.getSchemaMetadata().getValidationLevel());
}
Also used : SchemaMetadata(com.hortonworks.registries.schemaregistry.SchemaMetadata) SchemaVersion(com.hortonworks.registries.schemaregistry.SchemaVersion) SchemaFieldQuery(com.hortonworks.registries.schemaregistry.SchemaFieldQuery) SchemaIdVersion(com.hortonworks.registries.schemaregistry.SchemaIdVersion) SchemaVersionInfo(com.hortonworks.registries.schemaregistry.SchemaVersionInfo) SchemaVersionKey(com.hortonworks.registries.schemaregistry.SchemaVersionKey) SchemaMetadataInfo(com.hortonworks.registries.schemaregistry.SchemaMetadataInfo)

Example 15 with SchemaMetadata

use of com.hortonworks.registries.schemaregistry.SchemaMetadata in project registry by hortonworks.

the class SchemaBranchLifeCycleTest method addSchemaVersionToBranch.

@Test
public void addSchemaVersionToBranch() throws IOException, SchemaBranchNotFoundException, InvalidSchemaException, SchemaNotFoundException, IncompatibleSchemaException, SchemaBranchAlreadyExistsException {
    SchemaMetadata schemaMetadata = addSchemaMetadata(testNameRule.getMethodName(), SchemaCompatibility.NONE);
    SchemaIdVersion masterSchemaIdVersion1 = addSchemaVersion(SchemaBranch.MASTER_BRANCH, schemaMetadata, "/device.avsc");
    SchemaBranch schemaBranch1 = addSchemaBranch("BRANCH1", schemaMetadata, masterSchemaIdVersion1.getSchemaVersionId());
    addSchemaVersion(schemaBranch1.getName(), schemaMetadata, "/device-incompat.avsc");
    addSchemaVersion(schemaBranch1.getName(), schemaMetadata, "/device-compat.avsc");
    Collection<SchemaVersionInfo> schemaBranch1VersionInfos = schemaRegistryClient.getAllVersions(schemaBranch1.getName(), schemaMetadata.getName());
    Collection<SchemaVersionInfo> masterSchemaVersionInfos = schemaRegistryClient.getAllVersions(schemaMetadata.getName());
    Assert.assertTrue(masterSchemaVersionInfos.size() == 1);
    Assert.assertTrue(schemaBranch1VersionInfos.size() == 3);
    Long versionsInInitiatedState = schemaBranch1VersionInfos.stream().filter(schemaVersionInfo -> schemaVersionInfo.getStateId().equals(SchemaVersionLifecycleStates.INITIATED.getId())).count();
    Long versionsInEnabledState = schemaBranch1VersionInfos.stream().filter(schemaVersionInfo -> schemaVersionInfo.getStateId().equals(SchemaVersionLifecycleStates.ENABLED.getId())).count();
    Assert.assertTrue(versionsInInitiatedState == 2);
    Assert.assertTrue(versionsInEnabledState == 1);
}
Also used : Arrays(java.util.Arrays) SchemaVersion(com.hortonworks.registries.schemaregistry.SchemaVersion) SchemaBranchNotFoundException(com.hortonworks.registries.schemaregistry.errors.SchemaBranchNotFoundException) SchemaRegistryTestProfileType(com.hortonworks.registries.schemaregistry.avro.conf.SchemaRegistryTestProfileType) SchemaVersionLifecycleStates(com.hortonworks.registries.schemaregistry.state.SchemaVersionLifecycleStates) SchemaRegistryTestServerClientWrapper(com.hortonworks.registries.schemaregistry.avro.helper.SchemaRegistryTestServerClientWrapper) RunWith(org.junit.runner.RunWith) SchemaMetadata(com.hortonworks.registries.schemaregistry.SchemaMetadata) AvroSchemaRegistryClientUtil(com.hortonworks.registries.schemaregistry.avro.util.AvroSchemaRegistryClientUtil) HashSet(java.util.HashSet) Lists(com.google.common.collect.Lists) TestName(org.junit.rules.TestName) After(org.junit.After) SchemaVersionInfo(com.hortonworks.registries.schemaregistry.SchemaVersionInfo) Before(org.junit.Before) IncompatibleSchemaException(com.hortonworks.registries.schemaregistry.errors.IncompatibleSchemaException) SchemaCompatibility(com.hortonworks.registries.schemaregistry.SchemaCompatibility) InvalidSchemaException(com.hortonworks.registries.schemaregistry.errors.InvalidSchemaException) SetUtils(org.apache.commons.collections.SetUtils) Collection(java.util.Collection) InvalidSchemaBranchDeletionException(com.hortonworks.registries.schemaregistry.errors.InvalidSchemaBranchDeletionException) SchemaLifecycleException(com.hortonworks.registries.schemaregistry.state.SchemaLifecycleException) Set(java.util.Set) SchemaRegistryClient(com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient) Test(org.junit.Test) IOException(java.io.IOException) Collectors(java.util.stream.Collectors) SchemaBranch(com.hortonworks.registries.schemaregistry.SchemaBranch) Rule(org.junit.Rule) SchemaNotFoundException(com.hortonworks.registries.schemaregistry.errors.SchemaNotFoundException) CustomParameterizedRunner(com.hortonworks.registries.schemaregistry.avro.util.CustomParameterizedRunner) SchemaBranchAlreadyExistsException(com.hortonworks.registries.schemaregistry.errors.SchemaBranchAlreadyExistsException) Assert(org.junit.Assert) SchemaIdVersion(com.hortonworks.registries.schemaregistry.SchemaIdVersion) SchemaBranch(com.hortonworks.registries.schemaregistry.SchemaBranch) SchemaMetadata(com.hortonworks.registries.schemaregistry.SchemaMetadata) SchemaVersionInfo(com.hortonworks.registries.schemaregistry.SchemaVersionInfo) SchemaIdVersion(com.hortonworks.registries.schemaregistry.SchemaIdVersion) Test(org.junit.Test)

Aggregations

SchemaMetadata (com.hortonworks.registries.schemaregistry.SchemaMetadata)57 Test (org.junit.Test)35 SchemaIdVersion (com.hortonworks.registries.schemaregistry.SchemaIdVersion)33 SchemaVersion (com.hortonworks.registries.schemaregistry.SchemaVersion)31 SchemaVersionInfo (com.hortonworks.registries.schemaregistry.SchemaVersionInfo)21 SchemaBranch (com.hortonworks.registries.schemaregistry.SchemaBranch)17 SchemaMetadataInfo (com.hortonworks.registries.schemaregistry.SchemaMetadataInfo)15 SchemaVersionKey (com.hortonworks.registries.schemaregistry.SchemaVersionKey)11 SchemaNotFoundException (com.hortonworks.registries.schemaregistry.errors.SchemaNotFoundException)11 IncompatibleSchemaException (com.hortonworks.registries.schemaregistry.errors.IncompatibleSchemaException)9 InvalidSchemaException (com.hortonworks.registries.schemaregistry.errors.InvalidSchemaException)8 ByteArrayInputStream (java.io.ByteArrayInputStream)8 IOException (java.io.IOException)8 SchemaRegistryClient (com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient)7 IntegrationTest (com.hortonworks.registries.common.test.IntegrationTest)6 SchemaCompatibility (com.hortonworks.registries.schemaregistry.SchemaCompatibility)6 SchemaRegistryTestServerClientWrapper (com.hortonworks.registries.schemaregistry.avro.helper.SchemaRegistryTestServerClientWrapper)6 SchemaBranchNotFoundException (com.hortonworks.registries.schemaregistry.errors.SchemaBranchNotFoundException)6 Collection (java.util.Collection)6 SchemaBranchAlreadyExistsException (com.hortonworks.registries.schemaregistry.errors.SchemaBranchAlreadyExistsException)5