Search in sources :

Example 16 with SchemaField

use of org.apache.solr.schema.SchemaField in project lucene-solr by apache.

the class StatsCollectorSupplierFactory method buildFieldSource.

/**
   *  Builds a value source for a given field, making sure that the field fits a given source type.
   * @param schema the schema
   * @param expressionString The name of the field to build a Field Source from.
   * @param sourceType FIELD_TYPE for any type of field, NUMBER_TYPE for numeric fields, 
   * DATE_TYPE for date fields and STRING_TYPE for string fields.
   * @return a value source
   */
private static ValueSource buildFieldSource(IndexSchema schema, String expressionString, int sourceType) {
    SchemaField sf;
    try {
        sf = schema.getField(expressionString);
    } catch (SolrException e) {
        throw new SolrException(ErrorCode.BAD_REQUEST, "The field " + expressionString + " does not exist.", e);
    }
    FieldType type = sf.getType();
    if (type instanceof TrieIntField) {
        if (sourceType != NUMBER_TYPE && sourceType != FIELD_TYPE) {
            return null;
        }
        return new IntFieldSource(expressionString) {

            public String description() {
                return field;
            }
        };
    } else if (type instanceof TrieLongField) {
        if (sourceType != NUMBER_TYPE && sourceType != FIELD_TYPE) {
            return null;
        }
        return new LongFieldSource(expressionString) {

            public String description() {
                return field;
            }
        };
    } else if (type instanceof TrieFloatField) {
        if (sourceType != NUMBER_TYPE && sourceType != FIELD_TYPE) {
            return null;
        }
        return new FloatFieldSource(expressionString) {

            public String description() {
                return field;
            }
        };
    } else if (type instanceof TrieDoubleField) {
        if (sourceType != NUMBER_TYPE && sourceType != FIELD_TYPE) {
            return null;
        }
        return new DoubleFieldSource(expressionString) {

            public String description() {
                return field;
            }
        };
    } else if (type instanceof TrieDateField) {
        if (sourceType != DATE_TYPE && sourceType != FIELD_TYPE) {
            return null;
        }
        return new DateFieldSource(expressionString) {

            public String description() {
                return field;
            }
        };
    } else if (type instanceof StrField) {
        if (sourceType != STRING_TYPE && sourceType != FIELD_TYPE) {
            return null;
        }
        return new BytesRefFieldSource(expressionString) {

            public String description() {
                return field;
            }
        };
    }
    throw new SolrException(ErrorCode.BAD_REQUEST, type.toString() + " is not a supported field type in Solr Analytics.");
}
Also used : TrieDoubleField(org.apache.solr.schema.TrieDoubleField) StrField(org.apache.solr.schema.StrField) LongFieldSource(org.apache.lucene.queries.function.valuesource.LongFieldSource) TrieIntField(org.apache.solr.schema.TrieIntField) TrieDateField(org.apache.solr.schema.TrieDateField) BytesRefFieldSource(org.apache.lucene.queries.function.valuesource.BytesRefFieldSource) FieldType(org.apache.solr.schema.FieldType) SchemaField(org.apache.solr.schema.SchemaField) DoubleFieldSource(org.apache.lucene.queries.function.valuesource.DoubleFieldSource) IntFieldSource(org.apache.lucene.queries.function.valuesource.IntFieldSource) TrieLongField(org.apache.solr.schema.TrieLongField) FloatFieldSource(org.apache.lucene.queries.function.valuesource.FloatFieldSource) DateFieldSource(org.apache.solr.analytics.util.valuesource.DateFieldSource) SolrException(org.apache.solr.common.SolrException) TrieFloatField(org.apache.solr.schema.TrieFloatField)

Example 17 with SchemaField

use of org.apache.solr.schema.SchemaField in project lucene-solr by apache.

the class RangeEndpointCalculator method create.

public static RangeEndpointCalculator<? extends Comparable<?>> create(RangeFacetRequest request) {
    final SchemaField sf = request.getField();
    final FieldType ft = sf.getType();
    final RangeEndpointCalculator<?> calc;
    if (ft instanceof TrieField) {
        switch(ft.getNumberType()) {
            case FLOAT:
                calc = new FloatRangeEndpointCalculator(request);
                break;
            case DOUBLE:
                calc = new DoubleRangeEndpointCalculator(request);
                break;
            case INTEGER:
                calc = new IntegerRangeEndpointCalculator(request);
                break;
            case LONG:
                calc = new LongRangeEndpointCalculator(request);
                break;
            case DATE:
                calc = new DateRangeEndpointCalculator(request, null);
                break;
            default:
                throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unable to range facet on tried field of unexpected type:" + sf.getName());
        }
    } else {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unable to range facet on field:" + sf);
    }
    return calc;
}
Also used : SchemaField(org.apache.solr.schema.SchemaField) TrieField(org.apache.solr.schema.TrieField) SolrException(org.apache.solr.common.SolrException) FieldType(org.apache.solr.schema.FieldType)

Example 18 with SchemaField

use of org.apache.solr.schema.SchemaField in project lucene-solr by apache.

the class SolrQueryParserBase method getFieldQuery.

// if raw==true, then it's possible for this method to return a RawQuery that will need to be transformed
// further before using.
protected Query getFieldQuery(String field, String queryText, boolean quoted, boolean raw) throws SyntaxError {
    checkNullField(field);
    SchemaField sf;
    if (field.equals(lastFieldName)) {
        // only look up the SchemaField on a field change... this helps with memory allocation of dynamic fields
        // and large queries like foo_i:(1 2 3 4 5 6 7 8 9 10) when we are passed "foo_i" each time.
        sf = lastField;
    } else {
        // own functions.
        if (field.charAt(0) == '_' && parser != null) {
            MagicFieldName magic = MagicFieldName.get(field);
            if (null != magic) {
                subQParser = parser.subQuery(queryText, magic.subParser);
                return subQParser.getQuery();
            }
        }
        lastFieldName = field;
        sf = lastField = schema.getFieldOrNull(field);
    }
    if (sf != null) {
        FieldType ft = sf.getType();
        // delegate to type for everything except tokenized fields
        if (ft.isTokenized() && sf.indexed()) {
            boolean fieldAutoGenPhraseQueries = ft instanceof TextField && ((TextField) ft).getAutoGeneratePhraseQueries();
            boolean fieldEnableGraphQueries = ft instanceof TextField && ((TextField) ft).getEnableGraphQueries();
            return newFieldQuery(getAnalyzer(), field, queryText, quoted, fieldAutoGenPhraseQueries, fieldEnableGraphQueries);
        } else {
            if (raw) {
                return new RawQuery(sf, queryText);
            } else {
                return ft.getFieldQuery(parser, sf, queryText);
            }
        }
    }
    // default to a normal field query
    return newFieldQuery(getAnalyzer(), field, queryText, quoted, false, true);
}
Also used : SchemaField(org.apache.solr.schema.SchemaField) TextField(org.apache.solr.schema.TextField) FieldType(org.apache.solr.schema.FieldType)

Example 19 with SchemaField

use of org.apache.solr.schema.SchemaField in project lucene-solr by apache.

the class DocValuesFacets method getCounts.

public static NamedList<Integer> getCounts(SolrIndexSearcher searcher, DocSet docs, String fieldName, int offset, int limit, int mincount, boolean missing, String sort, String prefix, Predicate<BytesRef> termFilter, FacetDebugInfo fdebug) throws IOException {
    SchemaField schemaField = searcher.getSchema().getField(fieldName);
    FieldType ft = schemaField.getType();
    NamedList<Integer> res = new NamedList<>();
    // TODO: remove multiValuedFieldCache(), check dv type / uninversion type?
    final boolean multiValued = schemaField.multiValued() || ft.multiValuedFieldCache();
    // for term lookups only
    final SortedSetDocValues si;
    // for mapping per-segment ords to global ones
    OrdinalMap ordinalMap = null;
    if (multiValued) {
        si = searcher.getSlowAtomicReader().getSortedSetDocValues(fieldName);
        if (si instanceof MultiDocValues.MultiSortedSetDocValues) {
            ordinalMap = ((MultiSortedSetDocValues) si).mapping;
        }
    } else {
        SortedDocValues single = searcher.getSlowAtomicReader().getSortedDocValues(fieldName);
        si = single == null ? null : DocValues.singleton(single);
        if (single instanceof MultiDocValues.MultiSortedDocValues) {
            ordinalMap = ((MultiDocValues.MultiSortedDocValues) single).mapping;
        }
    }
    if (si == null) {
        return finalize(res, searcher, schemaField, docs, -1, missing);
    }
    if (si.getValueCount() >= Integer.MAX_VALUE) {
        throw new UnsupportedOperationException("Currently this faceting method is limited to " + Integer.MAX_VALUE + " unique terms");
    }
    final BytesRefBuilder prefixRef;
    if (prefix == null) {
        prefixRef = null;
    } else if (prefix.length() == 0) {
        prefix = null;
        prefixRef = null;
    } else {
        prefixRef = new BytesRefBuilder();
        prefixRef.copyChars(prefix);
    }
    int startTermIndex, endTermIndex;
    if (prefix != null) {
        startTermIndex = (int) si.lookupTerm(prefixRef.get());
        if (startTermIndex < 0)
            startTermIndex = -startTermIndex - 1;
        prefixRef.append(UnicodeUtil.BIG_TERM);
        endTermIndex = (int) si.lookupTerm(prefixRef.get());
        assert endTermIndex < 0;
        endTermIndex = -endTermIndex - 1;
    } else {
        startTermIndex = -1;
        endTermIndex = (int) si.getValueCount();
    }
    final int nTerms = endTermIndex - startTermIndex;
    int missingCount = -1;
    final CharsRefBuilder charsRef = new CharsRefBuilder();
    if (nTerms > 0 && docs.size() >= mincount) {
        // count collection array only needs to be as big as the number of terms we are
        // going to collect counts for.
        final int[] counts = new int[nTerms];
        if (fdebug != null) {
            fdebug.putInfoItem("numBuckets", nTerms);
        }
        Filter filter = docs.getTopFilter();
        List<LeafReaderContext> leaves = searcher.getTopReaderContext().leaves();
        for (int subIndex = 0; subIndex < leaves.size(); subIndex++) {
            LeafReaderContext leaf = leaves.get(subIndex);
            // solr docsets already exclude any deleted docs
            DocIdSet dis = filter.getDocIdSet(leaf, null);
            DocIdSetIterator disi = null;
            if (dis != null) {
                disi = dis.iterator();
            }
            if (disi != null) {
                if (multiValued) {
                    SortedSetDocValues sub = leaf.reader().getSortedSetDocValues(fieldName);
                    if (sub == null) {
                        sub = DocValues.emptySortedSet();
                    }
                    final SortedDocValues singleton = DocValues.unwrapSingleton(sub);
                    if (singleton != null) {
                        // some codecs may optimize SORTED_SET storage for single-valued fields
                        accumSingle(counts, startTermIndex, singleton, disi, subIndex, ordinalMap);
                    } else {
                        accumMulti(counts, startTermIndex, sub, disi, subIndex, ordinalMap);
                    }
                } else {
                    SortedDocValues sub = leaf.reader().getSortedDocValues(fieldName);
                    if (sub == null) {
                        sub = DocValues.emptySorted();
                    }
                    accumSingle(counts, startTermIndex, sub, disi, subIndex, ordinalMap);
                }
            }
        }
        if (startTermIndex == -1) {
            missingCount = counts[0];
        }
        // IDEA: we could also maintain a count of "other"... everything that fell outside
        // of the top 'N'
        int off = offset;
        int lim = limit >= 0 ? limit : Integer.MAX_VALUE;
        if (sort.equals(FacetParams.FACET_SORT_COUNT) || sort.equals(FacetParams.FACET_SORT_COUNT_LEGACY)) {
            int maxsize = limit > 0 ? offset + limit : Integer.MAX_VALUE - 1;
            maxsize = Math.min(maxsize, nTerms);
            LongPriorityQueue queue = new LongPriorityQueue(Math.min(maxsize, 1000), maxsize, Long.MIN_VALUE);
            // the smallest value in the top 'N' values
            int min = mincount - 1;
            for (int i = (startTermIndex == -1) ? 1 : 0; i < nTerms; i++) {
                int c = counts[i];
                if (c > min) {
                    if (termFilter != null) {
                        final BytesRef term = si.lookupOrd(startTermIndex + i);
                        if (!termFilter.test(term)) {
                            continue;
                        }
                    }
                    // smaller term numbers sort higher, so subtract the term number instead
                    long pair = (((long) c) << 32) + (Integer.MAX_VALUE - i);
                    boolean displaced = queue.insert(pair);
                    if (displaced)
                        min = (int) (queue.top() >>> 32);
                }
            }
            // if we are deep paging, we don't have to order the highest "offset" counts.
            int collectCount = Math.max(0, queue.size() - off);
            assert collectCount <= lim;
            // the start and end indexes of our list "sorted" (starting with the highest value)
            int sortedIdxStart = queue.size() - (collectCount - 1);
            int sortedIdxEnd = queue.size() + 1;
            final long[] sorted = queue.sort(collectCount);
            for (int i = sortedIdxStart; i < sortedIdxEnd; i++) {
                long pair = sorted[i];
                int c = (int) (pair >>> 32);
                int tnum = Integer.MAX_VALUE - (int) pair;
                final BytesRef term = si.lookupOrd(startTermIndex + tnum);
                ft.indexedToReadable(term, charsRef);
                res.add(charsRef.toString(), c);
            }
        } else {
            // add results in index order
            int i = (startTermIndex == -1) ? 1 : 0;
            if (mincount <= 0 && termFilter == null) {
                // if mincount<=0 and we're not examining the values for the term filter, then
                // we won't discard any terms and we know exactly where to start.
                i += off;
                off = 0;
            }
            for (; i < nTerms; i++) {
                int c = counts[i];
                if (c < mincount)
                    continue;
                BytesRef term = null;
                if (termFilter != null) {
                    term = si.lookupOrd(startTermIndex + i);
                    if (!termFilter.test(term)) {
                        continue;
                    }
                }
                if (--off >= 0)
                    continue;
                if (--lim < 0)
                    break;
                if (term == null) {
                    term = si.lookupOrd(startTermIndex + i);
                }
                ft.indexedToReadable(term, charsRef);
                res.add(charsRef.toString(), c);
            }
        }
    }
    return finalize(res, searcher, schemaField, docs, missingCount, missing);
}
Also used : MultiSortedSetDocValues(org.apache.lucene.index.MultiDocValues.MultiSortedSetDocValues) DocIdSet(org.apache.lucene.search.DocIdSet) MultiDocValues(org.apache.lucene.index.MultiDocValues) OrdinalMap(org.apache.lucene.index.MultiDocValues.OrdinalMap) LeafReaderContext(org.apache.lucene.index.LeafReaderContext) CharsRefBuilder(org.apache.lucene.util.CharsRefBuilder) BytesRef(org.apache.lucene.util.BytesRef) BytesRefBuilder(org.apache.lucene.util.BytesRefBuilder) LongPriorityQueue(org.apache.solr.util.LongPriorityQueue) NamedList(org.apache.solr.common.util.NamedList) SortedDocValues(org.apache.lucene.index.SortedDocValues) FieldType(org.apache.solr.schema.FieldType) SchemaField(org.apache.solr.schema.SchemaField) MultiSortedSetDocValues(org.apache.lucene.index.MultiDocValues.MultiSortedSetDocValues) SortedSetDocValues(org.apache.lucene.index.SortedSetDocValues) Filter(org.apache.solr.search.Filter) DocIdSetIterator(org.apache.lucene.search.DocIdSetIterator)

Example 20 with SchemaField

use of org.apache.solr.schema.SchemaField in project lucene-solr by apache.

the class DocValuesStats method getCounts.

public static StatsValues getCounts(SolrIndexSearcher searcher, StatsField statsField, DocSet docs, String[] facet) throws IOException {
    final SchemaField schemaField = statsField.getSchemaField();
    assert null != statsField.getSchemaField() : "DocValuesStats requires a StatsField using a SchemaField";
    final String fieldName = schemaField.getName();
    final FieldType ft = schemaField.getType();
    final StatsValues res = StatsValuesFactory.createStatsValues(statsField);
    //Initialize facetstats, if facets have been passed in
    final FieldFacetStats[] facetStats = new FieldFacetStats[facet.length];
    int upto = 0;
    for (String facetField : facet) {
        SchemaField fsf = searcher.getSchema().getField(facetField);
        if (fsf.multiValued()) {
            throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Stats can only facet on single-valued fields, not: " + facetField);
        }
        SchemaField facetSchemaField = searcher.getSchema().getField(facetField);
        facetStats[upto++] = new FieldFacetStats(searcher, facetSchemaField, statsField);
    }
    // TODO: remove multiValuedFieldCache(), check dv type / uninversion type?
    final boolean multiValued = schemaField.multiValued() || ft.multiValuedFieldCache();
    // for term lookups only
    SortedSetDocValues si;
    // for mapping per-segment ords to global ones
    OrdinalMap ordinalMap = null;
    if (multiValued) {
        si = searcher.getSlowAtomicReader().getSortedSetDocValues(fieldName);
        if (si instanceof MultiSortedSetDocValues) {
            ordinalMap = ((MultiDocValues.MultiSortedSetDocValues) si).mapping;
        }
    } else {
        SortedDocValues single = searcher.getSlowAtomicReader().getSortedDocValues(fieldName);
        si = single == null ? null : DocValues.singleton(single);
        if (single instanceof MultiDocValues.MultiSortedDocValues) {
            ordinalMap = ((MultiDocValues.MultiSortedDocValues) single).mapping;
        }
    }
    if (si == null) {
        si = DocValues.emptySortedSet();
    }
    if (si.getValueCount() >= Integer.MAX_VALUE) {
        throw new UnsupportedOperationException("Currently this stats method is limited to " + Integer.MAX_VALUE + " unique terms");
    }
    int missingDocCountTotal = 0;
    final int nTerms = (int) si.getValueCount();
    // count collection array only needs to be as big as the number of terms we are
    // going to collect counts for.
    final int[] counts = new int[nTerms];
    Filter filter = docs.getTopFilter();
    List<LeafReaderContext> leaves = searcher.getTopReaderContext().leaves();
    for (int subIndex = 0; subIndex < leaves.size(); subIndex++) {
        LeafReaderContext leaf = leaves.get(subIndex);
        // solr docsets already exclude any deleted docs
        DocIdSet dis = filter.getDocIdSet(leaf, null);
        DocIdSetIterator disi = null;
        if (dis != null) {
            disi = dis.iterator();
        }
        if (disi != null) {
            int docBase = leaf.docBase;
            if (multiValued) {
                SortedSetDocValues sub = leaf.reader().getSortedSetDocValues(fieldName);
                if (sub == null) {
                    sub = DocValues.emptySortedSet();
                }
                SortedDocValues singleton = DocValues.unwrapSingleton(sub);
                if (singleton != null) {
                    // some codecs may optimize SORTED_SET storage for single-valued fields
                    missingDocCountTotal += accumSingle(counts, docBase, facetStats, singleton, disi, subIndex, ordinalMap);
                } else {
                    missingDocCountTotal += accumMulti(counts, docBase, facetStats, sub, disi, subIndex, ordinalMap);
                }
            } else {
                SortedDocValues sub = leaf.reader().getSortedDocValues(fieldName);
                if (sub == null) {
                    sub = DocValues.emptySorted();
                }
                missingDocCountTotal += accumSingle(counts, docBase, facetStats, sub, disi, subIndex, ordinalMap);
            }
        }
    }
    // add results in index order
    for (int ord = 0; ord < counts.length; ord++) {
        int count = counts[ord];
        if (count > 0) {
            final BytesRef value = si.lookupOrd(ord);
            res.accumulate(value, count);
            for (FieldFacetStats f : facetStats) {
                f.accumulateTermNum(ord, value);
            }
        }
    }
    res.addMissing(missingDocCountTotal);
    if (facetStats.length > 0) {
        for (FieldFacetStats f : facetStats) {
            Map<String, StatsValues> facetStatsValues = f.facetStatsValues;
            f.accumulateMissing();
            res.addFacet(f.name, facetStatsValues);
        }
    }
    return res;
}
Also used : MultiSortedSetDocValues(org.apache.lucene.index.MultiDocValues.MultiSortedSetDocValues) DocIdSet(org.apache.lucene.search.DocIdSet) MultiDocValues(org.apache.lucene.index.MultiDocValues) OrdinalMap(org.apache.lucene.index.MultiDocValues.OrdinalMap) SortedDocValues(org.apache.lucene.index.SortedDocValues) FieldType(org.apache.solr.schema.FieldType) SchemaField(org.apache.solr.schema.SchemaField) FieldFacetStats(org.apache.solr.handler.component.FieldFacetStats) MultiSortedSetDocValues(org.apache.lucene.index.MultiDocValues.MultiSortedSetDocValues) SortedSetDocValues(org.apache.lucene.index.SortedSetDocValues) Filter(org.apache.solr.search.Filter) LeafReaderContext(org.apache.lucene.index.LeafReaderContext) StatsValues(org.apache.solr.handler.component.StatsValues) DocIdSetIterator(org.apache.lucene.search.DocIdSetIterator) SolrException(org.apache.solr.common.SolrException) BytesRef(org.apache.lucene.util.BytesRef)

Aggregations

SchemaField (org.apache.solr.schema.SchemaField)182 SolrException (org.apache.solr.common.SolrException)48 ArrayList (java.util.ArrayList)42 FieldType (org.apache.solr.schema.FieldType)41 IndexSchema (org.apache.solr.schema.IndexSchema)35 NamedList (org.apache.solr.common.util.NamedList)29 Query (org.apache.lucene.search.Query)23 IOException (java.io.IOException)22 BytesRef (org.apache.lucene.util.BytesRef)21 SimpleOrderedMap (org.apache.solr.common.util.SimpleOrderedMap)21 SolrIndexSearcher (org.apache.solr.search.SolrIndexSearcher)21 Document (org.apache.lucene.document.Document)20 SolrParams (org.apache.solr.common.params.SolrParams)19 IndexableField (org.apache.lucene.index.IndexableField)18 HashMap (java.util.HashMap)17 SolrInputDocument (org.apache.solr.common.SolrInputDocument)16 SolrDocument (org.apache.solr.common.SolrDocument)15 SolrQueryRequest (org.apache.solr.request.SolrQueryRequest)15 Map (java.util.Map)14 Term (org.apache.lucene.index.Term)14