HyperDex: Super-charging your NoSQL with a Strongly Consistent Datastore

I think it's weird when people believe there's a tool that will do their job for them, like a hammer that builds a roof by itself.

I'm sure HyperDex is totally useful for some cases, but it has clear disadvantages when you try to use it for what it wasn't intended (like global HP databases). All of a sudden you find yourself building glue to make it fit with your hybrid architecture. Instead you could take something simple and customize it, and build a huge successful business off of it, like the biggest sites in the world do currently with various tools that weren't engineered to solve simple problems like the number of round trips to look up an object.

Agreed with you fully that value-dependent chains are neat. They allow the system to replicate and relocate data, without any need for background processes. VDCs are the key to HyperDex's strong consistency guarantees.

With respect to "hyperspace hashing", it has been the standard way of interacting with large space decompositions for decades. The mathematics is from the 1960s (Morton et al) and the practical computer science was mature by the 1980s. When I was writing parallel codes for supercomputers, virtually all of our data structures were built this way. It is also the algorithm used for some large-scale geospatial systems; some of the Google Earth backends used it back in the day. While I've never personally worked on it, I've heard some graphics processing pipelines also use the algorithm. I first learned about it ten years ago because I ran into a couple systems using it.

I've never heard it called "hashing" until recently, largely because technically the algorithms are not a hash even if used as one in many implementations. This, and the fact that most of the literature on this idea significantly pre-dates electronic publishing, probably explains why you've seen so few references to hyper-dimensional hashing.

As you point out, the curse of dimensionality is a classic problem for these structures. The partitioning technique in the paper works for some use cases but is not used as much anymore. There have been some clever and fairly general attacks on the dimensionality problem over the last few years by a couple companies but I have not seen anything published.

All that said, this general approach is overdue for large-scale distributed systems. It has been proven out in niche markets like supercomputing for decades and is superior to distributed hash tables for many purposes. However, most of the recent research does not seem to be published.