Examples with DefaultTokenizer org.deeplearning4j.text.tokenization.tokenizer.DefaultTokenizer used on opensource projects

Example 1 with DefaultTokenizer

use of org.deeplearning4j.text.tokenization.tokenizer.DefaultTokenizer in project deeplearning4j by deeplearning4j.

the class DefaultTokenizerFactory method create.

@Override
public Tokenizer create(String toTokenize) {
    DefaultTokenizer t = new DefaultTokenizer(toTokenize);
    t.setTokenPreProcessor(tokenPreProcess);
    return t;
}