Examples with TaggedFileReader edu.stanford.nlp.tagger.io.TaggedFileReader used on opensource projects

Example 1 with TaggedFileReader

use of edu.stanford.nlp.tagger.io.TaggedFileReader in project CoreNLP by stanfordnlp.

the class CountClosedTags method countTrainingTags.

/**
   * Count trainingRatio of the sentences for both trainingWords and
   * allWords, and count the rest for just allWords
   */
void countTrainingTags(TaggedFileRecord file) throws IOException {
    int sentences = countSentences(file);
    int trainSentences = (int) (sentences * trainingRatio);
    TaggedFileReader reader = file.reader();
    List<TaggedWord> line;
    for (int i = 0; i < trainSentences && reader.hasNext(); ++i) {
        line = reader.next();
        addTaggedWords(line, trainingWords);
        addTaggedWords(line, allWords);
    }
    while (reader.hasNext()) {
        line = reader.next();
        addTaggedWords(line, allWords);
    }
}