Cache is the new RAM

I don't use Redis, but one thing I like about it is that it's a fairly generic storage layer that you can adapt into various data models; you can implement higher-level tools using its lower-level ones. I would love to be able to build a whole database machinery out of specific primitives.

For example, a "column" is basically a primitive you should be able to instantiate using some kind of storage strategy: keep it in RAM, keep it partitioned into chunks, append-only, compressed, sorted, remote etc.

The main thing that I want is to separate data from queries. It's always baffled me that traditional RDBMSes choose to commingle tables and indexes. Why does my commit have to wait for the indexes to catch up? Why do indexes have to be involved in the slow, complicated system of updating tuples that involve transactions, locking, paging and so on?

A better approach is to separate the two entirely:

1. The careful, slower ACID core that journals, fsyncs, keeps my data safe and provides a rigorous data model.

2. The super fast read-only, distributed indexes that organize the data in the most efficient way for querying (mostly through RAM and vectorization).

Now, I don't really need my write node to be a super-complicated distributed, replicated, gossip-based hash-ring-sharded, quorum-coordinated eventual consistency monstrosity. Single master is fine as long as you can keep standby replicas that are easy to fail over to.

What I want is for there to be a whole gaggle of read-only slave indexes that get my data reasonably quickly and are super fast to query. Indexes can be eventually consistent; but they don't need to be safe. Indexes can be reconstructed from the master data, after all. All we need to do is for indexes to ingest a continuous stream of data changes from the master.

Again, the master can be a bit slow: As slow as Postgres, at least. But the indexes, since they don't need to deal with transactions or locking or anything of the sort, can be really fast. Splitting the two up means they can each worry about different things, and apply different tolerances and constraints to what they do.

Today, we accomplish something similar by using ElasticSearch with Postgres. It's not good enough in the long run; they have completely different query mechanisms and data models, for one. It's also difficult to keep ES in perfect sync, and ES is generally heavy-weight; ES indexes are beasts, not very mobile. For example, the schema is mostly static, and changes require cloning the index (which I find weird). Still too fulltext-oriented; it's just not quite as good at non-text stuff. GIS support is lacking. And so on. A uniform query/database system is definitely needed.

SQL embeds in itself certain fundamental assumptions about operating so deep that you can't even see them. For instance, despite frequently being referred to as a "relational" query language, it requires some violations of the original relational logic. One example: The original relational logic looks a lot more like a tuple store where rows can freely contain arbitrary columns than the rigid row schema format that SQL so throughly assumes. For another, it has no concept of sharding, or indeed any other particular data-oriented structure of the database, which is simultaneously one of its strengths (it has survived precisely by not baking in such assumptions) but is also one of its weaknesses. If your database is sharded, for instance, there's nothing particularly in SQL itself that will guide you towards understanding whether you're writing a query that will or will not cross shards. You just have to "know". In fact there's a great deal of things about SQL performance that aren't reflected in the language and you just have to "know".

In theory, a new query language could potentially address these issues. In practice, what I've seen from "new" query languages has more to do with being "easy to implement" for the new store, and if that does or does not happen to meet any of these criteria, well wasn't that a happy coincidence. Someday I hope to see a well-considered sequel to SQL (no pun intended), but the way of thought that SQL affords has so thoroughly taken over the world that it is hard to see past it, much as it is hard to see past the von Neumann computer architecture into anything else. (I am not one of those people who think either von Neumann architecture or SQL are terrible and it all went wrong because we got stuck on them, but that's not to say they are the only way of doing business, either.)