Search in sources :

Example 6 with LanguageIdentifier

use of org.apache.tika.language.LanguageIdentifier in project lucene-solr by apache.

the class TikaLanguageIdentifierUpdateProcessor method detectLanguage.

@Override
protected List<DetectedLanguage> detectLanguage(SolrInputDocument doc) {
    List<DetectedLanguage> languages = new ArrayList<>();
    String content = concatFields(doc);
    if (content.length() != 0) {
        LanguageIdentifier identifier = new LanguageIdentifier(content);
        // FIXME: Hack - we get the distance from toString and calculate our own certainty score
        Double distance = Double.parseDouble(tikaSimilarityPattern.matcher(identifier.toString()).replaceFirst("$1"));
        // This formula gives: 0.02 => 0.8, 0.1 => 0.5 which is a better sweetspot than isReasonablyCertain()
        Double certainty = 1 - (5 * distance);
        if (certainty < 0)
            certainty = 0d;
        DetectedLanguage language = new DetectedLanguage(identifier.getLanguage(), certainty);
        languages.add(language);
        log.debug("Language detected as " + language + " with a certainty of " + language.getCertainty() + " (Tika distance=" + identifier.toString() + ")");
    } else {
        log.debug("No input text to detect language from, returning empty list");
    }
    return languages;
}
Also used : LanguageIdentifier(org.apache.tika.language.LanguageIdentifier) ArrayList(java.util.ArrayList)

Aggregations

LanguageIdentifier (org.apache.tika.language.LanguageIdentifier)6 IOException (java.io.IOException)2 Metadata (org.apache.tika.metadata.Metadata)2 Test (org.junit.Test)2 ParseException (edu.uci.ics.crawler4j.crawler.exceptions.ParseException)1 WebURL (edu.uci.ics.crawler4j.url.WebURL)1 ByteArrayInputStream (java.io.ByteArrayInputStream)1 InputStream (java.io.InputStream)1 UnsupportedEncodingException (java.io.UnsupportedEncodingException)1 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 Set (java.util.Set)1 Graph (org.apache.clerezza.commons.rdf.Graph)1 IRI (org.apache.clerezza.commons.rdf.IRI)1 PlainLiteralImpl (org.apache.clerezza.commons.rdf.impl.utils.PlainLiteralImpl)1 TripleImpl (org.apache.clerezza.commons.rdf.impl.utils.TripleImpl)1 Blob (org.apache.stanbol.enhancer.servicesapi.Blob)1 InvalidContentException (org.apache.stanbol.enhancer.servicesapi.InvalidContentException)1 ElasticsearchParseException (org.elasticsearch.ElasticsearchParseException)1