Search in sources :

Example 1 with TypeSignature

use of com.facebook.presto.spi.type.TypeSignature in project presto by prestodb.

the class HiveType method getTypeSignature.

@Nonnull
private static TypeSignature getTypeSignature(TypeInfo typeInfo) {
    switch(typeInfo.getCategory()) {
        case PRIMITIVE:
            Type primitiveType = getPrimitiveType((PrimitiveTypeInfo) typeInfo);
            if (primitiveType == null) {
                break;
            }
            return primitiveType.getTypeSignature();
        case MAP:
            MapTypeInfo mapTypeInfo = (MapTypeInfo) typeInfo;
            TypeSignature keyType = getTypeSignature(mapTypeInfo.getMapKeyTypeInfo());
            TypeSignature valueType = getTypeSignature(mapTypeInfo.getMapValueTypeInfo());
            return new TypeSignature(StandardTypes.MAP, ImmutableList.of(TypeSignatureParameter.of(keyType), TypeSignatureParameter.of(valueType)));
        case LIST:
            ListTypeInfo listTypeInfo = (ListTypeInfo) typeInfo;
            TypeSignature elementType = getTypeSignature(listTypeInfo.getListElementTypeInfo());
            return new TypeSignature(StandardTypes.ARRAY, ImmutableList.of(TypeSignatureParameter.of(elementType)));
        case STRUCT:
            StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo;
            List<TypeInfo> structFieldTypeInfos = structTypeInfo.getAllStructFieldTypeInfos();
            List<String> structFieldNames = structTypeInfo.getAllStructFieldNames();
            if (structFieldTypeInfos.size() != structFieldNames.size()) {
                throw new PrestoException(HiveErrorCode.HIVE_INVALID_METADATA, format("Invalid Hive struct type: %s", typeInfo));
            }
            ImmutableList.Builder<TypeSignatureParameter> typeSignatureBuilder = ImmutableList.builder();
            for (int i = 0; i < structFieldTypeInfos.size(); i++) {
                TypeSignature typeSignature = getTypeSignature(structFieldTypeInfos.get(i));
                // Lower case the struct field names.
                // Otherwise, Presto will refuse to write to columns whose struct type has field names containing upper case characters.
                // Users can't work around this by casting in their queries because Presto parser always lower case types.
                // TODO: This is a hack. Presto engine should be able to handle identifiers in a case insensitive way where necessary.
                String rowFieldName = structFieldNames.get(i).toLowerCase(Locale.US);
                typeSignatureBuilder.add(TypeSignatureParameter.of(new NamedTypeSignature(rowFieldName, typeSignature)));
            }
            return new TypeSignature(StandardTypes.ROW, typeSignatureBuilder.build());
    }
    throw new PrestoException(NOT_SUPPORTED, format("Unsupported Hive type: %s", typeInfo));
}
Also used : ImmutableList(com.google.common.collect.ImmutableList) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) StructTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo) PrestoException(com.facebook.presto.spi.PrestoException) TypeInfoUtils.getTypeInfosFromTypeString(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString) TypeInfoUtils.getTypeInfoFromTypeString(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfoFromTypeString) TypeInfoFactory.doubleTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.doubleTypeInfo) TypeInfoFactory.dateTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.dateTypeInfo) MapTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo) StructTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo) PrimitiveTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo) TypeInfoFactory.longTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.longTypeInfo) TypeInfoFactory.shortTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.shortTypeInfo) VarcharTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.VarcharTypeInfo) TypeInfoFactory.timestampTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.timestampTypeInfo) TypeInfoFactory.floatTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.floatTypeInfo) TypeInfoFactory.intTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.intTypeInfo) TypeInfoFactory.binaryTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.binaryTypeInfo) TypeInfoFactory.byteTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.byteTypeInfo) ListTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo) TypeInfoFactory.booleanTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.booleanTypeInfo) DecimalTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo) TypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfo) TypeInfoFactory.stringTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory.stringTypeInfo) CharTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.CharTypeInfo) CharType.createCharType(com.facebook.presto.spi.type.CharType.createCharType) VarcharType.createVarcharType(com.facebook.presto.spi.type.VarcharType.createVarcharType) DecimalType.createDecimalType(com.facebook.presto.spi.type.DecimalType.createDecimalType) Type(com.facebook.presto.spi.type.Type) VarcharType.createUnboundedVarcharType(com.facebook.presto.spi.type.VarcharType.createUnboundedVarcharType) TypeSignature(com.facebook.presto.spi.type.TypeSignature) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) TypeSignatureParameter(com.facebook.presto.spi.type.TypeSignatureParameter) ListTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo) MapTypeInfo(org.apache.hadoop.hive.serde2.typeinfo.MapTypeInfo) Nonnull(javax.annotation.Nonnull)

Example 2 with TypeSignature

use of com.facebook.presto.spi.type.TypeSignature in project presto by prestodb.

the class QueryResults method fixValue.

/**
     * Force values coming from Jackson to have the expected object type.
     */
private static Object fixValue(TypeSignature signature, Object value) {
    if (value == null) {
        return null;
    }
    if (signature.getBase().equals(ARRAY)) {
        List<Object> fixedValue = new ArrayList<>();
        for (Object object : List.class.cast(value)) {
            fixedValue.add(fixValue(signature.getTypeParametersAsTypeSignatures().get(0), object));
        }
        return fixedValue;
    }
    if (signature.getBase().equals(MAP)) {
        TypeSignature keySignature = signature.getTypeParametersAsTypeSignatures().get(0);
        TypeSignature valueSignature = signature.getTypeParametersAsTypeSignatures().get(1);
        Map<Object, Object> fixedValue = new HashMap<>();
        for (Map.Entry<?, ?> entry : (Set<Map.Entry<?, ?>>) Map.class.cast(value).entrySet()) {
            fixedValue.put(fixValue(keySignature, entry.getKey()), fixValue(valueSignature, entry.getValue()));
        }
        return fixedValue;
    }
    if (signature.getBase().equals(ROW)) {
        Map<String, Object> fixedValue = new LinkedHashMap<>();
        List<Object> listValue = List.class.cast(value);
        checkArgument(listValue.size() == signature.getParameters().size(), "Mismatched data values and row type");
        for (int i = 0; i < listValue.size(); i++) {
            TypeSignatureParameter parameter = signature.getParameters().get(i);
            checkArgument(parameter.getKind() == ParameterKind.NAMED_TYPE, "Unexpected parameter [%s] for row type", parameter);
            NamedTypeSignature namedTypeSignature = parameter.getNamedTypeSignature();
            String key = namedTypeSignature.getName();
            fixedValue.put(key, fixValue(namedTypeSignature.getTypeSignature(), listValue.get(i)));
        }
        return fixedValue;
    }
    switch(signature.getBase()) {
        case BIGINT:
            if (value instanceof String) {
                return Long.parseLong((String) value);
            }
            return ((Number) value).longValue();
        case INTEGER:
            if (value instanceof String) {
                return Integer.parseInt((String) value);
            }
            return ((Number) value).intValue();
        case SMALLINT:
            if (value instanceof String) {
                return Short.parseShort((String) value);
            }
            return ((Number) value).shortValue();
        case TINYINT:
            if (value instanceof String) {
                return Byte.parseByte((String) value);
            }
            return ((Number) value).byteValue();
        case DOUBLE:
            if (value instanceof String) {
                return Double.parseDouble((String) value);
            }
            return ((Number) value).doubleValue();
        case REAL:
            if (value instanceof String) {
                return Float.parseFloat((String) value);
            }
            return ((Number) value).floatValue();
        case BOOLEAN:
            if (value instanceof String) {
                return Boolean.parseBoolean((String) value);
            }
            return Boolean.class.cast(value);
        case VARCHAR:
        case JSON:
        case TIME:
        case TIME_WITH_TIME_ZONE:
        case TIMESTAMP:
        case TIMESTAMP_WITH_TIME_ZONE:
        case DATE:
        case INTERVAL_YEAR_TO_MONTH:
        case INTERVAL_DAY_TO_SECOND:
        case DECIMAL:
        case CHAR:
            return String.class.cast(value);
        default:
            // as a plain text and everything else is base64 encoded binary
            if (value instanceof String) {
                return Base64.getDecoder().decode((String) value);
            }
            return value;
    }
}
Also used : Set(java.util.Set) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) ArrayList(java.util.ArrayList) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) LinkedHashMap(java.util.LinkedHashMap) TypeSignature(com.facebook.presto.spi.type.TypeSignature) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) TypeSignature.parseTypeSignature(com.facebook.presto.spi.type.TypeSignature.parseTypeSignature) TypeSignatureParameter(com.facebook.presto.spi.type.TypeSignatureParameter) HashMap(java.util.HashMap) LinkedHashMap(java.util.LinkedHashMap) Map(java.util.Map)

Example 3 with TypeSignature

use of com.facebook.presto.spi.type.TypeSignature in project presto by prestodb.

the class MongoSession method guessFieldType.

private Optional<TypeSignature> guessFieldType(Object value) {
    if (value == null) {
        return Optional.empty();
    }
    TypeSignature typeSignature = null;
    if (value instanceof String) {
        typeSignature = createUnboundedVarcharType().getTypeSignature();
    } else if (value instanceof Integer || value instanceof Long) {
        typeSignature = BIGINT.getTypeSignature();
    } else if (value instanceof Boolean) {
        typeSignature = BOOLEAN.getTypeSignature();
    } else if (value instanceof Float || value instanceof Double) {
        typeSignature = DOUBLE.getTypeSignature();
    } else if (value instanceof Date) {
        typeSignature = TIMESTAMP.getTypeSignature();
    } else if (value instanceof ObjectId) {
        typeSignature = OBJECT_ID.getTypeSignature();
    } else if (value instanceof List) {
        List<Optional<TypeSignature>> subTypes = ((List<?>) value).stream().map(this::guessFieldType).collect(toList());
        if (subTypes.isEmpty() || subTypes.stream().anyMatch(t -> !t.isPresent())) {
            return Optional.empty();
        }
        Set<TypeSignature> signatures = subTypes.stream().map(t -> t.get()).collect(toSet());
        if (signatures.size() == 1) {
            typeSignature = new TypeSignature(StandardTypes.ARRAY, signatures.stream().map(s -> TypeSignatureParameter.of(s)).collect(Collectors.toList()));
        } else {
            // TODO: presto cli doesn't handle empty field name row type yet
            typeSignature = new TypeSignature(StandardTypes.ROW, IntStream.range(0, subTypes.size()).mapToObj(idx -> TypeSignatureParameter.of(new NamedTypeSignature(String.format("%s%d", implicitPrefix, idx + 1), subTypes.get(idx).get()))).collect(toList()));
        }
    } else if (value instanceof Document) {
        List<TypeSignatureParameter> parameters = new ArrayList<>();
        for (String key : ((Document) value).keySet()) {
            Optional<TypeSignature> fieldType = guessFieldType(((Document) value).get(key));
            if (!fieldType.isPresent()) {
                return Optional.empty();
            }
            parameters.add(TypeSignatureParameter.of(new NamedTypeSignature(key, fieldType.get())));
        }
        typeSignature = new TypeSignature(StandardTypes.ROW, parameters);
    }
    return Optional.ofNullable(typeSignature);
}
Also used : Document(org.bson.Document) Arrays(java.util.Arrays) LoadingCache(com.google.common.cache.LoadingCache) TypeManager(com.facebook.presto.spi.type.TypeManager) Date(java.util.Date) MongoDatabase(com.mongodb.client.MongoDatabase) BIGINT(com.facebook.presto.spi.type.BigintType.BIGINT) SchemaTableName(com.facebook.presto.spi.SchemaTableName) BOOLEAN(com.facebook.presto.spi.type.BooleanType.BOOLEAN) SchemaNotFoundException(com.facebook.presto.spi.SchemaNotFoundException) Map(java.util.Map) TypeSignatureParameter(com.facebook.presto.spi.type.TypeSignatureParameter) StandardTypes(com.facebook.presto.spi.type.StandardTypes) Collectors.toSet(java.util.stream.Collectors.toSet) ImmutableSet(com.google.common.collect.ImmutableSet) Set(java.util.Set) Collectors(java.util.stream.Collectors) Preconditions.checkState(com.google.common.base.Preconditions.checkState) CacheLoader(com.google.common.cache.CacheLoader) TupleDomain(com.facebook.presto.spi.predicate.TupleDomain) Domain(com.facebook.presto.spi.predicate.Domain) List(java.util.List) FindIterable(com.mongodb.client.FindIterable) Optional(java.util.Optional) CacheBuilder(com.google.common.cache.CacheBuilder) TypeSignature(com.facebook.presto.spi.type.TypeSignature) IntStream(java.util.stream.IntStream) DOUBLE(com.facebook.presto.spi.type.DoubleType.DOUBLE) MongoCollection(com.mongodb.client.MongoCollection) Slice(io.airlift.slice.Slice) Logger(io.airlift.log.Logger) OBJECT_ID(com.facebook.presto.mongodb.ObjectIdType.OBJECT_ID) MINUTES(java.util.concurrent.TimeUnit.MINUTES) ArrayList(java.util.ArrayList) HashSet(java.util.HashSet) ImmutableList(com.google.common.collect.ImmutableList) MongoCursor(com.mongodb.client.MongoCursor) Verify.verify(com.google.common.base.Verify.verify) Type(com.facebook.presto.spi.type.Type) UncheckedExecutionException(com.google.common.util.concurrent.UncheckedExecutionException) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) Objects.requireNonNull(java.util.Objects.requireNonNull) TIMESTAMP(com.facebook.presto.spi.type.TimestampType.TIMESTAMP) Throwables(com.google.common.base.Throwables) Range(com.facebook.presto.spi.predicate.Range) IndexOptions(com.mongodb.client.model.IndexOptions) ExecutionException(java.util.concurrent.ExecutionException) VarcharType.createUnboundedVarcharType(com.facebook.presto.spi.type.VarcharType.createUnboundedVarcharType) Collectors.toList(java.util.stream.Collectors.toList) TableNotFoundException(com.facebook.presto.spi.TableNotFoundException) ColumnHandle(com.facebook.presto.spi.ColumnHandle) MongoClient(com.mongodb.MongoClient) DeleteResult(com.mongodb.client.result.DeleteResult) ObjectId(org.bson.types.ObjectId) HOURS(java.util.concurrent.TimeUnit.HOURS) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Optional(java.util.Optional) ObjectId(org.bson.types.ObjectId) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) Document(org.bson.Document) Date(java.util.Date) TypeSignature(com.facebook.presto.spi.type.TypeSignature) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) List(java.util.List) ArrayList(java.util.ArrayList) ImmutableList(com.google.common.collect.ImmutableList) Collectors.toList(java.util.stream.Collectors.toList)

Example 4 with TypeSignature

use of com.facebook.presto.spi.type.TypeSignature in project presto by prestodb.

the class MongoSession method guessTableFields.

private List<Document> guessTableFields(SchemaTableName schemaTableName) {
    String schemaName = schemaTableName.getSchemaName();
    String tableName = schemaTableName.getTableName();
    MongoDatabase db = client.getDatabase(schemaName);
    Document doc = db.getCollection(tableName).find().first();
    if (doc == null) {
        // no records at the collection
        return ImmutableList.of();
    }
    ImmutableList.Builder<Document> builder = ImmutableList.builder();
    for (String key : doc.keySet()) {
        Object value = doc.get(key);
        Optional<TypeSignature> fieldType = guessFieldType(value);
        if (fieldType.isPresent()) {
            Document metadata = new Document();
            metadata.append(FIELDS_NAME_KEY, key);
            metadata.append(FIELDS_TYPE_KEY, fieldType.get().toString());
            metadata.append(FIELDS_HIDDEN_KEY, key.equals("_id") && fieldType.get().equals(OBJECT_ID.getTypeSignature()));
            builder.add(metadata);
        } else {
            log.debug("Unable to guess field type from %s : %s", value == null ? "null" : value.getClass().getName(), value);
        }
    }
    return builder.build();
}
Also used : TypeSignature(com.facebook.presto.spi.type.TypeSignature) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) ImmutableList(com.google.common.collect.ImmutableList) Document(org.bson.Document) MongoDatabase(com.mongodb.client.MongoDatabase)

Example 5 with TypeSignature

use of com.facebook.presto.spi.type.TypeSignature in project presto by prestodb.

the class OrcStorageManager method getType.

private Type getType(List<OrcType> types, int index) {
    OrcType type = types.get(index);
    switch(type.getOrcTypeKind()) {
        case BOOLEAN:
            return BOOLEAN;
        case LONG:
            return BIGINT;
        case DOUBLE:
            return DOUBLE;
        case STRING:
            return createUnboundedVarcharType();
        case BINARY:
            return VARBINARY;
        case DECIMAL:
            return DecimalType.createDecimalType(type.getPrecision().get(), type.getScale().get());
        case LIST:
            TypeSignature elementType = getType(types, type.getFieldTypeIndex(0)).getTypeSignature();
            return typeManager.getParameterizedType(StandardTypes.ARRAY, ImmutableList.of(TypeSignatureParameter.of(elementType)));
        case MAP:
            TypeSignature keyType = getType(types, type.getFieldTypeIndex(0)).getTypeSignature();
            TypeSignature valueType = getType(types, type.getFieldTypeIndex(1)).getTypeSignature();
            return typeManager.getParameterizedType(StandardTypes.MAP, ImmutableList.of(TypeSignatureParameter.of(keyType), TypeSignatureParameter.of(valueType)));
        case STRUCT:
            List<String> fieldNames = type.getFieldNames();
            ImmutableList.Builder<TypeSignatureParameter> fieldTypes = ImmutableList.builder();
            for (int i = 0; i < type.getFieldCount(); i++) {
                fieldTypes.add(TypeSignatureParameter.of(new NamedTypeSignature(fieldNames.get(i), getType(types, type.getFieldTypeIndex(i)).getTypeSignature())));
            }
            return typeManager.getParameterizedType(StandardTypes.ROW, fieldTypes.build());
    }
    throw new PrestoException(RAPTOR_ERROR, "Unhandled ORC type: " + type);
}
Also used : TypeSignature(com.facebook.presto.spi.type.TypeSignature) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) OrcType(com.facebook.presto.orc.metadata.OrcType) TypeSignatureParameter(com.facebook.presto.spi.type.TypeSignatureParameter) ImmutableList(com.google.common.collect.ImmutableList) NamedTypeSignature(com.facebook.presto.spi.type.NamedTypeSignature) PrestoException(com.facebook.presto.spi.PrestoException)

Aggregations

TypeSignature (com.facebook.presto.spi.type.TypeSignature)34 TypeSignature.parseTypeSignature (com.facebook.presto.spi.type.TypeSignature.parseTypeSignature)19 ImmutableList (com.google.common.collect.ImmutableList)14 NamedTypeSignature (com.facebook.presto.spi.type.NamedTypeSignature)10 Test (org.testng.annotations.Test)10 Signature (com.facebook.presto.metadata.Signature)9 PrestoException (com.facebook.presto.spi.PrestoException)8 Type (com.facebook.presto.spi.type.Type)6 TypeSignatureParameter (com.facebook.presto.spi.type.TypeSignatureParameter)6 ImmutableSet (com.google.common.collect.ImmutableSet)6 Slice (io.airlift.slice.Slice)6 UsedByGeneratedCode (com.facebook.presto.annotation.UsedByGeneratedCode)5 SCALAR (com.facebook.presto.metadata.FunctionKind.SCALAR)5 SqlScalarFunction (com.facebook.presto.metadata.SqlScalarFunction)5 SqlScalarFunctionBuilder (com.facebook.presto.metadata.SqlScalarFunctionBuilder)5 SpecializeContext (com.facebook.presto.metadata.SqlScalarFunctionBuilder.SpecializeContext)5 SqlType (com.facebook.presto.spi.function.SqlType)5 DecimalType (com.facebook.presto.spi.type.DecimalType)5 Decimals (com.facebook.presto.spi.type.Decimals)5 Decimals.encodeUnscaledValue (com.facebook.presto.spi.type.Decimals.encodeUnscaledValue)5