Search in sources :

Example 6 with SchemaParseException

use of org.apache.avro.SchemaParseException in project flink by apache.

the class GlueSchemaRegistryInputStreamDeserializer method getSchemaAndDeserializedStream.

/**
 * Get schema and remove extra Schema Registry information within input stream.
 *
 * @param in input stream
 * @return schema of object within input stream
 * @throws IOException Exception during decompression
 */
public Schema getSchemaAndDeserializedStream(InputStream in) throws IOException {
    byte[] inputBytes = new byte[in.available()];
    in.read(inputBytes);
    in.reset();
    MutableByteArrayInputStream mutableByteArrayInputStream = (MutableByteArrayInputStream) in;
    String schemaDefinition = glueSchemaRegistryDeserializationFacade.getSchemaDefinition(inputBytes);
    byte[] deserializedBytes = glueSchemaRegistryDeserializationFacade.getActualData(inputBytes);
    mutableByteArrayInputStream.setBuffer(deserializedBytes);
    Schema schema;
    try {
        Parser schemaParser = new Schema.Parser();
        schema = schemaParser.parse(schemaDefinition);
    } catch (SchemaParseException e) {
        String message = "Error occurred while parsing schema, see inner exception for details.";
        throw new AWSSchemaRegistryException(message, e);
    }
    return schema;
}
Also used : AWSSchemaRegistryException(com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException) MutableByteArrayInputStream(org.apache.flink.formats.avro.utils.MutableByteArrayInputStream) SchemaParseException(org.apache.avro.SchemaParseException) Schema(org.apache.avro.Schema) Parser(org.apache.avro.Schema.Parser)

Example 7 with SchemaParseException

use of org.apache.avro.SchemaParseException in project incubator-gobblin by apache.

the class HiveMetaStoreUtils method getDeserializer.

/**
 * Returns a Deserializer from HiveRegistrationUnit if present and successfully initialized. Else returns null.
 */
private static Deserializer getDeserializer(HiveRegistrationUnit unit) {
    Optional<String> serdeClass = unit.getSerDeType();
    if (!serdeClass.isPresent()) {
        return null;
    }
    String serde = serdeClass.get();
    HiveConf hiveConf;
    Deserializer deserializer;
    try {
        hiveConf = SharedResourcesBrokerFactory.getImplicitBroker().getSharedResource(new HiveConfFactory<>(), SharedHiveConfKey.INSTANCE);
        deserializer = ReflectionUtils.newInstance(hiveConf.getClassByName(serde).asSubclass(Deserializer.class), hiveConf);
    } catch (ClassNotFoundException e) {
        LOG.warn("Serde class " + serde + " not found!", e);
        return null;
    } catch (NotConfiguredException nce) {
        LOG.error("Implicit broker is not configured properly", nce);
        return null;
    }
    Properties props = new Properties();
    props.putAll(unit.getProps().getProperties());
    props.putAll(unit.getStorageProps().getProperties());
    props.putAll(unit.getSerDeProps().getProperties());
    try {
        SerDeUtils.initializeSerDe(deserializer, hiveConf, props, null);
        // handling in AvroSerDe added in HIVE-7868.
        if (deserializer instanceof AvroSerDe) {
            try {
                inVokeDetermineSchemaOrThrowExceptionMethod(props, new Configuration());
            } catch (SchemaParseException | InvocationTargetException | NoSuchMethodException | IllegalAccessException e) {
                LOG.warn("Failed to initialize AvroSerDe.");
                throw new SerDeException(e);
            }
        }
    } catch (SerDeException e) {
        LOG.warn("Failed to initialize serde " + serde + " with properties " + props + " for table " + unit.getDbName() + "." + unit.getTableName());
        return null;
    }
    return deserializer;
}
Also used : NotConfiguredException(org.apache.gobblin.broker.iface.NotConfiguredException) Configuration(org.apache.hadoop.conf.Configuration) HiveConfFactory(org.apache.gobblin.hive.HiveConfFactory) Properties(java.util.Properties) InvocationTargetException(java.lang.reflect.InvocationTargetException) AvroSerDe(org.apache.hadoop.hive.serde2.avro.AvroSerDe) Deserializer(org.apache.hadoop.hive.serde2.Deserializer) SchemaParseException(org.apache.avro.SchemaParseException) HiveConf(org.apache.hadoop.hive.conf.HiveConf) SerDeException(org.apache.hadoop.hive.serde2.SerDeException)

Aggregations

SchemaParseException (org.apache.avro.SchemaParseException)7 Schema (org.apache.avro.Schema)4 AvroRowDeserializationSchema (org.apache.flink.formats.avro.AvroRowDeserializationSchema)2 AvroRowSerializationSchema (org.apache.flink.formats.avro.AvroRowSerializationSchema)2 UnexpectedFormatException (co.cask.cdap.api.data.format.UnexpectedFormatException)1 UnsupportedTypeException (co.cask.cdap.api.data.schema.UnsupportedTypeException)1 AWSSchemaRegistryException (com.amazonaws.services.schemaregistry.exception.AWSSchemaRegistryException)1 File (java.io.File)1 IOException (java.io.IOException)1 InvocationTargetException (java.lang.reflect.InvocationTargetException)1 NoSuchAlgorithmException (java.security.NoSuchAlgorithmException)1 ArrayList (java.util.ArrayList)1 Properties (java.util.Properties)1 Parser (org.apache.avro.Schema.Parser)1 TypeInformation (org.apache.flink.api.common.typeinfo.TypeInformation)1 MutableByteArrayInputStream (org.apache.flink.formats.avro.utils.MutableByteArrayInputStream)1 NotConfiguredException (org.apache.gobblin.broker.iface.NotConfiguredException)1 HiveConfFactory (org.apache.gobblin.hive.HiveConfFactory)1 Configuration (org.apache.hadoop.conf.Configuration)1 HiveConf (org.apache.hadoop.hive.conf.HiveConf)1