Live data from Hacker News

Hitchhiker trees: functional, persistent, off-heap sorted maps

github.com

11–20 of 33 posts

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#11
post #9

Does someone know of an existing visualization of this algorithm, for better understanding?

Creator here. There's no visualizations of this yet, but I'll be speaking about it this year at Strange Loop. For that talk, I'll be creating visualizations, so stay tuned!

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#12
post #5

Funny: I implemented the exact same thing, with a focus on AWS and pagination (albeit less well-documented): https://github.com/rix0rrr/libbruce/blob/master/README.md

That is very cool! When you end up doing reads, do you compute all the pending writes along the path from root to leaf, and then run queries on the projection?

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#14

Is it possible to use this from Java directly, or does the Clojure API make this difficult?

Creator here. You could use this from Java using the Clojure API for Java; however, many of the extension hooks use Clojure protocols. I would recommend writing a shim to Java for your particular application, so that your data types are represented in the way you want.

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#15
post #14

Is it possible to use this from Java directly, or does the Clojure API make this difficult?

Creator here. You could use this from Java using the Clojure API for Java; however, many of the extension hooks use Clojure protocols. I would recommend writing a shim to Java for your particular application, so that your data types are represented in the way you want.

That'd required the person doing the using to know Clojure, right?

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#16
> the best-performing data structure for looking up sorted keys cannot do those queries faster than O(log(n))

Feels like a digression off the actual tree structure, but isn't this incorrect, since a hash map can do key lookups in O(1)? With caveats, of course.

It's a great idea over all, but I'd also be curious how well overlaying an event log on a cuckoo hash would work in comparison.

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#17

> the best-performing data structure for looking up sorted keys cannot do those queries faster than O(log(n)) Feels like a digression off the actual tree structure, but isn't this incorrect, since a hash map can do key lookups in O(1)? With caveats, of course. It's a great idea over all, but I'd also be curious how well overlaying an event log on a cuckoo hash would work in comparison.

keyword here is sorted. hash map is unordered

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#18
post #12
post #5

Funny: I implemented the exact same thing, with a focus on AWS and pagination (albeit less well-documented): https://github.com/rix0rrr/libbruce/blob/master/README.md

That is very cool! When you end up doing reads, do you compute all the pending writes along the path from root to leaf, and then run queries on the projection?

For index-seeks, yes.

For key-seeks, it's a normal tree walk to the leaf (unless an INSERT is found at a higher level, iirc)

In practice, the blocks are so huge that any tree is going to have at most depth 3, and the root page is likely in cache, so 1 or 2 s3 fetches (= roughly 200ms, let's say)

I've been toying with the idea of putting the root page (or maybe the top 2 levels of pages) in DynamoDB, which would make them fast even if they weren't in cache.

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#19

> the best-performing data structure for looking up sorted keys cannot do those queries faster than O(log(n)) Feels like a digression off the actual tree structure, but isn't this incorrect, since a hash map can do key lookups in O(1)? With caveats, of course. It's a great idea over all, but I'd also be curious how well overlaying an event log on a cuckoo hash would work in comparison.

A hashmap isn't sorted keys. But 'O(log(n))' is still incorrect. The fastest you can look up information physically represented in a 3D universe is 'O(cube-root(N))'. That applies to radom access memory and hashmaps, too. Cf. Myth of RAM [1].

[1] http://www.ilikebigbits.com/blog/2014/4/21/the-myth-of-ram-p...

Re: Hitchhiker trees: functional, persistent, off-heap sorted maps

#20
post #14

Earlier quoted context omitted.

Creator here. You could use this from Java using the Clojure API for Java; however, many of the extension hooks use Clojure protocols. I would recommend writing a shim to Java for your particular application, so that your data types are represented in the way you want.

That'd required the person doing the using to know Clojure, right?

Indeed that would.
Post reply on HN