Examples with NGramTokenizer org.deeplearning4j.text.tokenization.tokenizer.NGramTokenizer used on opensource projects

Example 1 with NGramTokenizer

use of org.deeplearning4j.text.tokenization.tokenizer.NGramTokenizer in project deeplearning4j by deeplearning4j.

the class NGramTokenizerFactory method create.

@Override
public Tokenizer create(String toTokenize) {
    if (toTokenize == null || toTokenize.isEmpty()) {
        throw new IllegalArgumentException("Unable to proceed; no sentence to tokenize");
    }
    Tokenizer t1 = tokenizerFactory.create(toTokenize);
    t1.setTokenPreProcessor(preProcess);
    Tokenizer ret = new NGramTokenizer(t1, minN, maxN);
    return ret;
}