Search in sources :

Example 1 with StringTypeEncoded

use of com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded in project conquery by bakdata.

the class MapTypeGuesser method createGuess.

@Override
public Guess createGuess() {
    IntegerStore indexType = p.decideIndexType();
    StringTypeDictionary type = new StringTypeDictionary(indexType, null);
    long mapSize = MapDictionary.estimateMemoryConsumption(p.getStrings().size(), p.getDecoded().stream().mapToLong(s -> s.length).sum());
    StringTypeEncoded result = new StringTypeEncoded(type, p.getEncoding());
    return new Guess(result, indexType.estimateMemoryConsumptionBytes(), mapSize) {

        @Override
        public StringStore getType() {
            MapDictionary map = new MapDictionary(Dataset.PLACEHOLDER, UUID.randomUUID().toString());
            for (byte[] v : p.getDecoded()) {
                map.add(v);
            }
            type.setDictionary(map);
            return super.getType();
        }
    };
}
Also used : IntegerStore(com.bakdata.conquery.models.events.stores.root.IntegerStore) StringTypeDictionary(com.bakdata.conquery.models.events.stores.specific.string.StringTypeDictionary) StringTypeEncoded(com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded) MapDictionary(com.bakdata.conquery.models.dictionary.MapDictionary)

Example 2 with StringTypeEncoded

use of com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded in project conquery by bakdata.

the class TrieTypeGuesser method createGuess.

@Override
public Guess createGuess() {
    IntegerStore indexType = p.decideIndexType();
    SuccinctTrie trie = new SuccinctTrie(Dataset.PLACEHOLDER, UUID.randomUUID().toString());
    StringTypeDictionary type = new StringTypeDictionary(indexType, trie);
    for (byte[] v : p.getDecoded()) {
        trie.add(v);
    }
    StringTypeEncoded result = new StringTypeEncoded(type, p.getEncoding());
    return new Guess(result, indexType.estimateMemoryConsumptionBytes(), trie.estimateMemoryConsumption()) {

        @Override
        public StringStore getType() {
            trie.compress();
            return super.getType();
        }
    };
}
Also used : IntegerStore(com.bakdata.conquery.models.events.stores.root.IntegerStore) StringTypeDictionary(com.bakdata.conquery.models.events.stores.specific.string.StringTypeDictionary) StringTypeEncoded(com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded) SuccinctTrie(com.bakdata.conquery.util.dict.SuccinctTrie)

Example 3 with StringTypeEncoded

use of com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded in project conquery by bakdata.

the class StringEncodingTest method testHexStreamStringType.

@Test
public void testHexStreamStringType() {
    StringParser parser = new StringParser(new ConqueryConfig());
    Stream.generate(() -> UUID.randomUUID().toString().replace("-", "")).map(String::toUpperCase).mapToInt(v -> {
        try {
            return parser.parse(v);
        } catch (ParsingException e) {
            // We know that StringTypeVarInt is able to parse our strings.
            return 0;
        }
    }).limit(100).forEach(parser::addLine);
    StringTypeEncoded subType = (StringTypeEncoded) parser.findBestType();
    assertThat(subType).isInstanceOf(StringTypeEncoded.class);
    assertThat(subType.getEncoding()).isEqualByComparingTo(StringTypeEncoded.Encoding.Base16UpperCase);
}
Also used : StringParser(com.bakdata.conquery.models.preproc.parser.specific.StringParser) StringTypeEncoded(com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded) ParsingException(com.bakdata.conquery.models.exceptions.ParsingException) ConqueryConfig(com.bakdata.conquery.models.config.ConqueryConfig) Test(org.junit.jupiter.api.Test) DynamicTest(org.junit.jupiter.api.DynamicTest)

Aggregations

StringTypeEncoded (com.bakdata.conquery.models.events.stores.specific.string.StringTypeEncoded)3 IntegerStore (com.bakdata.conquery.models.events.stores.root.IntegerStore)2 StringTypeDictionary (com.bakdata.conquery.models.events.stores.specific.string.StringTypeDictionary)2 ConqueryConfig (com.bakdata.conquery.models.config.ConqueryConfig)1 MapDictionary (com.bakdata.conquery.models.dictionary.MapDictionary)1 ParsingException (com.bakdata.conquery.models.exceptions.ParsingException)1 StringParser (com.bakdata.conquery.models.preproc.parser.specific.StringParser)1 SuccinctTrie (com.bakdata.conquery.util.dict.SuccinctTrie)1 DynamicTest (org.junit.jupiter.api.DynamicTest)1 Test (org.junit.jupiter.api.Test)1