Live data from Hacker News

Crit-bit Trees: the best way to store sets

cr.yp.to

41–42 of 42 posts

Re: Crit-bit Trees: the best way to store sets

#41
post #40
post #16

> Another advantage is that a crit-bit tree guarantees good performance: it doesn't have any tricky slowdowns for unusual (or malicious) data. Consider a crit-bit tree over variable-length bit strings. The set 1, 11, 111, ... (with n bit strings) has depth n. Depending on how you count, it can be argued that this is still substantially better than the worst-case of hash tables, but in all of the ways I think of count…

Big O notation, it isn't. The big question is the cost of different types of comparisons. The most important thing to remember is that a crit-bit lookup does O(string-length) "critical" bit lookups, and then an O(string-length) final comparison - for a total of O(string-length). Whereas a hash lookup usually does O(string-length) computation of the hash value, average case O(1) lookups, and then O(string-length) comp…

Good points.

Given a query string of length k and cache lines that store B bits, the crit-bit trie for the bad input uses Theta(k) computations and causes Theta(k) cache misses, I think. A balanced tree holding the same set causes Theta(k * lg n) computations and O(lg n + (k * lg n / B)) cache misses, I think. My understanding is that lg n / B is frequently much less than 1, even for small cache line sizes, so that the balanced tree will generally use less I/O than a crit-bit tree but more computation.

Re: Crit-bit Trees: the best way to store sets

#42
post #32
post #21

Crit-bit trees are great but they're not silver-bullet: like every tree structure, even in the lucky case a search can cause O(log n) cache misses, while a good hash table needs just one or two. Also, as pointed out in another comment, it is not necessarily balanced (like red-black trees) and there are some real-world datasets where crit-bit (or Patricia) trees behave poorly, for example URLs. I myself have seen Patr…

That's a very good point. One way to circumvent this problem is to combine hashing and crit-bit trees, by constructing an unordered tree on the hash values of the set elements. The nice thing here is that if the hash function behaves like a uniform random variable the resulting tree will be balanced. Furthermore, without ordering the implementation is trivial. At the same time you will not need rehashing once the tab…

The string b-tree also does trie-like search without sacrificing balance:

http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.57.59...

Post reply on HN