Examples with Tokenizer org.apache.hadoop.hbase.codec.prefixtree.encode.tokenize.Tokenizer used on opensource projects

Example 1 with Tokenizer

use of org.apache.hadoop.hbase.codec.prefixtree.encode.tokenize.Tokenizer in project hbase by apache.

the class PrefixTreeEncoder method initializeTagHelpers.

protected void initializeTagHelpers() {
    this.tagsRange = new SimpleMutableByteRange();
    this.tagsDeduplicator = USE_HASH_COLUMN_SORTER ? new ByteRangeHashSet() : new ByteRangeTreeSet();
    this.tagsTokenizer = new Tokenizer();
    this.tagsWriter = new ColumnSectionWriter();
}

Example 2 with Tokenizer

use of org.apache.hadoop.hbase.codec.prefixtree.encode.tokenize.Tokenizer in project hbase by apache.

the class TestTreeDepth method testInternal.

protected void testInternal(List<String> inputs, int expectedTreeDepth) {
    Tokenizer builder = new Tokenizer();
    for (String s : inputs) {
        SimpleMutableByteRange b = new SimpleMutableByteRange(Bytes.toBytes(s));
        builder.addSorted(b);
    }
    Assert.assertEquals(1, builder.getRoot().getNodeDepth());
    Assert.assertEquals(expectedTreeDepth, builder.getTreeDepth());
}