RethinkDB: why we failed (2017)

A question that lacks a satisfactory answer in open source data infrastructure is this: who is going to pay for the initial architecture to be state-of-the-art? Architecture is forever, if you start with a naive design it will put the platform at a long-term disadvantage. Most open source infrastructure projects are started by well-meaning people that have no idea what a state-of-the-art design even looks like. At the periphery you might be able to burn a giant pile of VC money to make it happen but that is not sustainable without a clear path to profitable exit. The set of people with the critical expertise and the set of people building open source data infrastructure are nearly disjoint.

Today, the money is in state-of-the-art bespoke data infrastructure. If the tradeoff is "open source" and "10-100x better on infrastructure KPIs", companies happily pay millions for the latter. People that know how to design this infrastructure are extremely well paid, 7-figures is common and demand is high. Yes, they could take a year off work and create an open source infrastructure project but this doesn't seem to happen and it is easy to understand why, as there are few incentives and many disincentives. This is roughly my area of business, I see the dynamics from the inside and the trend is not favoring open source.

If all of the cutting edge infrastructure development work is happening in closed source, whether cloud or state-of-the-art bespoke, it doesn't bode well for the long-term relevance of open source.

As a former RethinkDB user (we have migrated to Postgres) I actually don't miss it as much as I would - JSONB in PG does what we need, and the real-time features of RethinkDB never really delivered because of various performance issues in the database itself.

This article has less to do with databases or open source software than it does with the fundamentally misaligned incentives of egalitarianism and capitalism.

I wish there was a social open source license that said "you can use this indefinitely as long as you pay us something". The price would be up to the user but the generally accepted polite minimum would be at least a penny ($0.01).

Then businesses could publicly display their level of support (both initially and yearly) for the software they use, to attract customers. They could also be audited by the IRS, so a lack of patronage could correlate to a lack of equality and maybe even reveal corruption and other malfeasance. At the very least it would reveal a lack of internal controls.

This could work kind of like UBI for open source. And I'm definitely not the only one who has ever thought about this. But there is just so much free capital floating around right now that it's a great time to be thinking about how to reform the paradigms that we all depend on.

I agree with you on redis-like utilities being better for pure KV stores. If you resort to that sort of thing regularly and want to give another look into postgres you might want to give Elixir a try. It has first class support for postgres with Ecto, and you can use redis-like stores from the language itself, skipping serialization (a note is that, one of the valuable things in redis usually comes from being in a separate instance, many times managed automatically, and so almost being a permanent cache - to have exactly the same properties you would need to dig a bit deeper in Elixir, but still pretty doable with the language building blocks).