Examples with JcsegTokenizer com.jstarcraft.nlp.tokenization.jcseg.JcsegTokenizer used on opensource projects

Example 1 with JcsegTokenizer

use of com.jstarcraft.nlp.tokenization.jcseg.JcsegTokenizer in project jstarcraft-nlp by HongZhaoHua.

the class JcsegTokenizerTestCase method getTokenizer.

@Override
protected NlpTokenizer<? extends NlpToken> getTokenizer() {
    try {
        // 自动查找jcseg.properties配置
        JcsegTaskConfig configuration = new JcsegTaskConfig(true);
        // 默认单例词库
        ADictionary dictionary = DictionaryFactory.createSingletonDictionary(configuration);
        // 依据给定JcsegTaskConfig和ADictionary构建ISegment
        ISegment segment = SegmentFactory.createJcseg(JcsegTaskConfig.COMPLEX_MODE, new Object[] { configuration, dictionary });
        return new JcsegTokenizer(segment);
    } catch (Exception exception) {
        throw new RuntimeException(exception);
    }
}

Example 2 with JcsegTokenizer

use of com.jstarcraft.nlp.tokenization.jcseg.JcsegTokenizer in project jstarcraft-nlp by HongZhaoHua.

the class JcsegSegmentFactory method getNlpTokenizer.

@Override
protected NlpTokenizer<? extends NlpToken> getNlpTokenizer(Map<String, String> configurations) {
    ISegment segment = build(configurations);
    JcsegTokenizer tokenizer = new JcsegTokenizer(segment);
    return tokenizer;
}