Examples with TurkishLexer zemberek.tokenization.antlr.TurkishLexer used on opensource projects

Example 1 with TurkishLexer

use of zemberek.tokenization.antlr.TurkishLexer in project zemberek-nlp by ahmetaa.

the class TurkishTokenizer method lexerInstance.

private static TurkishLexer lexerInstance(CharStream inputStream) {
    TurkishLexer lexer = new TurkishLexer(inputStream);
    lexer.removeErrorListeners();
    lexer.addErrorListener(IGNORING_ERROR_LISTENER);
    return lexer;
}