HyperDex: Super-charging your NoSQL with a Strongly Consistent Datastore

Bypassing the content, I love the idea of having a suggested soundtrack for a blog post! - along with the link to actually hear it :-)

I was wondering how long it would take for HyperDex to get some traction on HN. While I haven't had immediate need to use it myself, I have several friends who attest that it is a lovely system. The article doesn't mention it, but it's wicked fast, outperforming Redis in many metrics. Clearly the consistency is the big kicker though. So many systems are saying eventual consistency is "good enough." Clearly that can be fine for a SoLoMo app, but probably not for a medical system.

Lack of consistency guarantees in most systems available in the open is the exact reason why we created Arakoon: our usage scenarios (safekeeping metadata for large-scale storage systems, among others) simply don't allow for 'eventual consistency'. It's also available as free software, check it out at http://arakoon.org.

NoSQL/SQL trolling aside, doing data lookup in a hyperspace is such a great idea. Being able to slice my data on hyperplanes is such a useful and cool feature.

In fact, database theory was originally conceived as hyper-dimensional metric space in which both data and selection could be mapped to hyper-rectangles in the space. A hyper-plane or a point in the metric space is just a degenerate hyper-rectangle with the same dimensionality as the metric space. It has always been the most elegant way to represent a database.

One of the best written expositions of databases properly expressed as operations over hyper-dimensional spaces was produced by Rudolf Bayer, the guy that invented the B-Tree. In the late 1990s he invented a multidimensional indexing structure based on space-filling curves called a UB-Tree and they wrote at length about how various database operations are implemented using that representation. It is generally informative if you are unfamiliar with this aspect of database theory, not just in the context of UB-Trees about which it was written.

If it is such a great idea then why does no major database implement things this way? Despite several attempts by companies like Oracle, IBM, and Microsoft, no one has described a generalized data structure for databases with hyper-rectangle operands as your primitives. There are dozens of narrow algorithm solutions known, both published and unpublished, but none that you could legitimately use in a commercial database system because they all have limitations that will adversely affect real-world applications.

Hyperdex is a conventional algorithm from the standpoint of indexing hyper-dimensional spaces. They are not doing anything new there that has not been done before. However, the update value chaining element of it is actually pretty neat.