Examples with MmsegTokenizer com.jstarcraft.nlp.tokenization.mmseg.MmsegTokenizer used on opensource projects

Example 1 with MmsegTokenizer

use of com.jstarcraft.nlp.tokenization.mmseg.MmsegTokenizer in project jstarcraft-nlp by HongZhaoHua.

the class MmsegSegmentFactory method getNlpTokenizer.

@Override
protected NlpTokenizer<? extends NlpToken> getNlpTokenizer(Map<String, String> configurations) {
    MMSeg segment = build(configurations);
    MmsegTokenizer tokenizer = new MmsegTokenizer(segment);
    return tokenizer;
}

Example 2 with MmsegTokenizer

use of com.jstarcraft.nlp.tokenization.mmseg.MmsegTokenizer in project jstarcraft-nlp by HongZhaoHua.

the class MmsegTokenizerTestCase method getTokenizer.

@Override
protected NlpTokenizer<? extends NlpToken> getTokenizer() {
    Dictionary dictionary = Dictionary.getInstance();
    ComplexSeg complex = new ComplexSeg(dictionary);
    MMSeg mmSeg = new MMSeg(new StringReader(""), complex);
    return new MmsegTokenizer(mmSeg);
}