Search in sources :

Example 46 with MappingMetaData

use of org.elasticsearch.cluster.metadata.MappingMetaData in project metron by apache.

the class ElasticsearchColumnMetadataDao method getColumnMetadata.

@SuppressWarnings("unchecked")
@Override
public Map<String, FieldType> getColumnMetadata(List<String> indices) throws IOException {
    Map<String, FieldType> indexColumnMetadata = new HashMap<>();
    Map<String, String> previousIndices = new HashMap<>();
    Set<String> fieldBlackList = new HashSet<>();
    String[] latestIndices = getLatestIndices(indices);
    if (latestIndices.length > 0) {
        ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>> mappings = adminClient.indices().getMappings(new GetMappingsRequest().indices(latestIndices)).actionGet().getMappings();
        // for each index
        for (Object key : mappings.keys().toArray()) {
            String indexName = key.toString();
            ImmutableOpenMap<String, MappingMetaData> mapping = mappings.get(indexName);
            // for each mapping in the index
            Iterator<String> mappingIterator = mapping.keysIt();
            while (mappingIterator.hasNext()) {
                MappingMetaData mappingMetaData = mapping.get(mappingIterator.next());
                Map<String, Object> sourceAsMap = mappingMetaData.getSourceAsMap();
                if (sourceAsMap.containsKey("properties")) {
                    Map<String, Map<String, String>> map = (Map<String, Map<String, String>>) sourceAsMap.get("properties");
                    // for each field in the mapping
                    for (String field : map.keySet()) {
                        if (!fieldBlackList.contains(field)) {
                            FieldType type = toFieldType(map.get(field).get("type"));
                            if (!indexColumnMetadata.containsKey(field)) {
                                indexColumnMetadata.put(field, type);
                                // record the last index in which a field exists, to be able to print helpful error message on type mismatch
                                previousIndices.put(field, indexName);
                            } else {
                                FieldType previousType = indexColumnMetadata.get(field);
                                if (!type.equals(previousType)) {
                                    String previousIndexName = previousIndices.get(field);
                                    LOG.error(String.format("Field type mismatch: %s.%s has type %s while %s.%s has type %s.  Defaulting type to %s.", indexName, field, type.getFieldType(), previousIndexName, field, previousType.getFieldType(), FieldType.OTHER.getFieldType()));
                                    indexColumnMetadata.put(field, FieldType.OTHER);
                                    // the field is defined in multiple indices with different types; ignore the field as type has been set to OTHER
                                    fieldBlackList.add(field);
                                }
                            }
                        }
                    }
                }
            }
        }
    } else {
        LOG.info(String.format("Unable to find any latest indices; indices=%s", indices));
    }
    return indexColumnMetadata;
}
Also used : HashMap(java.util.HashMap) ImmutableOpenMap(org.elasticsearch.common.collect.ImmutableOpenMap) MappingMetaData(org.elasticsearch.cluster.metadata.MappingMetaData) GetMappingsRequest(org.elasticsearch.action.admin.indices.mapping.get.GetMappingsRequest) FieldType(org.apache.metron.indexing.dao.search.FieldType) ImmutableOpenMap(org.elasticsearch.common.collect.ImmutableOpenMap) HashMap(java.util.HashMap) Map(java.util.Map) HashSet(java.util.HashSet)

Aggregations

MappingMetaData (org.elasticsearch.cluster.metadata.MappingMetaData)46 ImmutableOpenMap (org.elasticsearch.common.collect.ImmutableOpenMap)16 Map (java.util.Map)15 GetMappingsResponse (org.elasticsearch.action.admin.indices.mapping.get.GetMappingsResponse)11 IndexMetaData (org.elasticsearch.cluster.metadata.IndexMetaData)10 Settings (org.elasticsearch.common.settings.Settings)9 Test (org.junit.Test)9 IOException (java.io.IOException)8 HashMap (java.util.HashMap)7 IndexTemplateMetaData (org.elasticsearch.cluster.metadata.IndexTemplateMetaData)7 PartitionName (io.crate.metadata.PartitionName)6 BytesRef (org.apache.lucene.util.BytesRef)6 SearchResponse (org.elasticsearch.action.search.SearchResponse)5 Set (java.util.Set)4 ClusterState (org.elasticsearch.cluster.ClusterState)4 CompressedXContent (org.elasticsearch.common.compress.CompressedXContent)4 ObjectObjectCursor (com.carrotsearch.hppc.cursors.ObjectObjectCursor)3 ArrayList (java.util.ArrayList)3 List (java.util.List)3 GetIndexTemplatesResponse (org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse)3