Examples with UimaTokenizerFactory org.deeplearning4j.text.tokenization.tokenizerfactory.UimaTokenizerFactory used on opensource projects

Example 1 with UimaTokenizerFactory

use of org.deeplearning4j.text.tokenization.tokenizerfactory.UimaTokenizerFactory in project deeplearning4j by deeplearning4j.

the class Word2VecIteratorTest method before.

@Before
public void before() throws Exception {
    if (vec == null) {
        ClassPathResource resource = new ClassPathResource("/labeled/");
        File file = resource.getFile();
        SentenceIterator iter = UimaSentenceIterator.createWithPath(file.getAbsolutePath());
        new File("cache.ser").delete();
        TokenizerFactory t = new UimaTokenizerFactory();
        vec = new Word2Vec.Builder().minWordFrequency(1).iterations(5).layerSize(100).stopWords(new ArrayList<String>()).useUnknown(true).windowSize(5).iterate(iter).tokenizerFactory(t).build();
        vec.fit();
    }
}