Examples with WordTokenizer org.languagetool.tokenizers.WordTokenizer used on opensource projects

Example 11 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class SlovakTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new SlovakTagger();
    tokenizer = new WordTokenizer();
}

Example 12 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class SpanishTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new SpanishTagger();
    tokenizer = new WordTokenizer();
}

Example 13 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class FrenchTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new FrenchTagger();
    tokenizer = new WordTokenizer();
}

Example 14 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class GalicianTaggerTest method setUp.

@Before
public void setUp() {
    tagger = new GalicianTagger();
    tokenizer = new WordTokenizer();
}

Example 15 with WordTokenizer

use of org.languagetool.tokenizers.WordTokenizer in project languagetool by languagetool-org.

the class PolishDisambiguationRuleTest method setUp.

@Before
public void setUp() {
    tagger = new PolishTagger();
    tokenizer = new WordTokenizer();
    sentenceTokenizer = new SRXSentenceTokenizer(new Polish());
    disambiguator = new MultiWordChunker("/pl/multiwords.txt");
}