Examples with Stemming com.yahoo.searchdefinition.document.Stemming used on opensource projects

Example 1 with Stemming

use of com.yahoo.searchdefinition.document.Stemming in project vespa by vespa-engine.

the class TextMatch method findAnnotatorConfig.

private AnnotatorConfig findAnnotatorConfig(Search search, SDField field) {
    AnnotatorConfig ret = new AnnotatorConfig();
    Stemming activeStemming = field.getStemming();
    if (activeStemming == null) {
        activeStemming = search.getStemming();
    }
    ret.setStemMode(activeStemming.toStemMode());
    ret.setRemoveAccents(field.getNormalizing().doRemoveAccents());
    if ((field.getMatching() != null) && (field.getMatching().maxLength() != null)) {
        ret.setMaxTokenLength(field.getMatching().maxLength());
    }
    return ret;
}

Example 2 with Stemming

use of com.yahoo.searchdefinition.document.Stemming in project vespa by vespa-engine.

the class StemmingResolver method checkStemmingForIndexFields.

private void checkStemmingForIndexFields(String indexName, List<SDField> fields) {
    Stemming stemming = null;
    SDField stemmingField = null;
    for (SDField field : fields) {
        if (stemming == null && stemmingField == null) {
            stemming = field.getStemming(search);
            stemmingField = field;
        } else if (stemming != field.getStemming(search)) {
            deployLogger.log(Level.WARNING, "Field '" + field.getName() + "' has " + field.getStemming(search) + ", whereas field '" + stemmingField.getName() + "' has " + stemming + ". All fields indexing to the index '" + indexName + "' must have the same stemming." + " This should be corrected as it will make indexing fail in a few cases.");
        }
    }
}

Example 3 with Stemming

use of com.yahoo.searchdefinition.document.Stemming in project vespa by vespa-engine.

the class FieldSetValidity method checkStemming.

private void checkStemming(Search search, FieldSet fieldSet) {
    Stemming fsStemming = null;
    for (String fld : fieldSet.getFieldNames()) {
        ImmutableSDField field = search.getField(fld);
        Stemming fieldStemming = field.getStemming();
        if (fsStemming == null) {
            fsStemming = fieldStemming;
        } else {
            if (!fsStemming.equals(fieldStemming)) {
                warn(search, field.asField(), "The stemming settings for the fields in the fieldset '" + fieldSet.getName() + "' are inconsistent (explicitly or because of field type). " + "This may lead to recall and ranking issues.");
                return;
            }
        }
    }
}

Example 4 with Stemming

use of com.yahoo.searchdefinition.document.Stemming in project vespa by vespa-engine.

the class IndexingScriptChangeMessageBuilder method checkStemming.

private void checkStemming(ChangeMessageBuilder builder) {
    Stemming currentStemming = currentField.getStemming(currentSearch);
    Stemming nextStemming = nextField.getStemming(nextSearch);
    if (!currentStemming.equals(nextStemming)) {
        builder.addChange("stemming", currentStemming.getName(), nextStemming.getName());
    }
}