Examples with WordTokenizer org.languagetool.tokenizers.WordTokenizer used on opensource projects

Example 6 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class FrenchRuleDisambiguatorTest method setUp.

@Before
public void setUp() throws IOException {
    tagger = new FrenchTagger();
    tokenizer = new WordTokenizer();
    French language = new French();
    sentenceTokenizer = new SRXSentenceTokenizer(language);
    disambiguator = new XmlRuleDisambiguator(language);
    disamb2 = new DemoDisambiguator();
}

Example 7 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class EsperantoTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new EsperantoTagger();
    tokenizer = new WordTokenizer();
}

Example 8 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class AbstractRomanianTaggerTest method setUp.

@Before
public void setUp() {
    tagger = createTagger();
    tokenizer = new WordTokenizer();
}

Example 9 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class PortugueseTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new PortugueseTagger();
    tokenizer = new WordTokenizer();
}

Example 10 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class RussianTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new RussianTagger();
    tokenizer = new WordTokenizer();
}