Examples with StrTokenizer org.apache.commons.lang3.text.StrTokenizer used on opensource projects

Example 1 with StrTokenizer

use of org.apache.commons.lang3.text.StrTokenizer in project ASCIIGenome by dariober.

the class Utils method tokenize.

/**
 * Split string x in tokens. Effectively just a friendly wrapper around StrTokenizer.
 * Use *single* quotes for avoiding splitting.
 */
public static ArrayList<String> tokenize(String x, String delimiterString) {
    if (x == null) {
        return null;
    }
    // This is a hack to allow empty tokens to be passed at the command line.
    // An empty
    x = x.replace("''", "' '");
    // See also http://stackoverflow.com/questions/38161437/inconsistent-behaviour-of-strtokenizer-to-split-string
    StrTokenizer str = new StrTokenizer(x);
    str.setTrimmerMatcher(StrMatcher.spaceMatcher());
    str.setDelimiterString(delimiterString);
    str.setQuoteChar('\'');
    // str.setIgnoreEmptyTokens(false);
    ArrayList<String> tokens = (ArrayList<String>) str.getTokenList();
    for (int i = 0; i < tokens.size(); i++) {
        String tok = tokens.get(i).trim();
        tokens.set(i, tok);
    }
    return tokens;
}

Example 2 with StrTokenizer

use of org.apache.commons.lang3.text.StrTokenizer in project apps-android-wikipedia by wikimedia.

the class CsvColumn method tokenizer.

@NonNull
private StrTokenizer tokenizer(@Nullable String str) {
    StrTokenizer tokenizer = StrTokenizer.getCSVInstance(str);
    tokenizer.setTrimmerMatcher(StrMatcher.noneMatcher());
    tokenizer.setEmptyTokenAsNull(true);
    return tokenizer;
}