Immutability, MVCC, and garbage collection

Pretty disappointing critique.

There are manifold differences between couchdb, datomic, rethinkdb and more traditional sql databases, but the author can't see past his pet issue. He doesn't seem to understand the use cases, or the infrastructure differences. In terms of use cases, there are plenty of analytically datasets that are strictly monotonic. There is no opportunity to reclaim overwritten storage in this case.

But CoW index write amplification you say? Well now let's talk about infrastructure differences. Datomic can use s3 as a storage layer. Cold data storage is then $10/month/terabyte. And compaction is trivially non-blocking in a CoW dataset. It can be done by EC2 spec instances whenver is convenient with zero impact on the production system. Don't generalize the experiences of a few folks running couchdb on a couple in house servers to people working with a fundamentally different approach on fundamentally different infrastructure.

Back to the cheap cost of cold storage, for most businesses that's well into "who cares" cost territory, but having the data also allows them to preserve full lineage in the advent they discover a data loss bug in their application. That could potentially be quite valuable. One way to think of these systems is that they have integrated backup data. MySQL folks generally don't count the size of backup sets as part of the database footprint, and common backup schedules (eg father, grandfather) provide only a limited ability to dig back into the past.

By way of simple argument: would anyone advise using a source control system that only kept the last few hundreds of commits? Why do we treat our data differently? The answer is largely one of implementation constraints: text is cheap, vcs doesn't index the way databases do, etc. But as storage and processing densities continue to grow more datasets shift to the trivial category. As much as many HNews posters dream of being in a 'big data' shop, the vast majority of startups have maybe a couple GB in MySQL or Postgres. Our status quo toolset and approach is discarding value for these folks. Maybe you think that's fine, but some of us are going to keep pushing to enable new capabilities and advantages. Increasingly these implementation constraints will disappear as people attack them. It'd be nice to not suffer scornful surface criticisms from people who base their career around the status quo.

"If entities are made of related facts, and some facts are updated but others aren’t, then as the database grows and new facts are recorded, an entity ends up being scattered widely over a lot of storage."

As far as I understand this isn't so. All data in Datomic is stored in the indices, so the facts associated with an entity should be stored together.

I think you are making a mistake by reading it as a critique of immutability in general. Rather, I think the author is criticizing an overly-simplisitic approach to using immutability.

When he describes traditional databases:

"What’s happening in such a database is a combination of short-term immutability, read and write optimizations to save and/or coalesce redundant work, and continuous “compaction” and reuse of disk space to stabilize disk usage and avoid infinite growth."

He's saying that's what people arrive at after working out the issues with immutability. When you say: "Increasingly these implementation constraints will disappear as people attack them." the author would probably respond: "Yes, and people will attack them with a lot of the same strategies that SQL databases are already using today".

If we are allowing the appeal to authority fallacy then neither would Baron Schwartz write this article out of ignorance. We have two different authorities disagreeing here.

Append-only b-tree/skiplist/whatever immutability can be combined with "checkpoint hashes" (think git) using for example SHA512, that can be transmitted elsewhere at a low cost and used later to validate system integrity. Any data store tampering would be detectable. In a mutable database you'd need to transmit whole transaction log for the same integrity guarantee.

Thus immutable store makes it trivial to implement systems that need to have full history and audit trail.

When there's only a trivial amount of data involved, immutable state can solve a lot of headaches. No programming error can lose or corrupt data for instance - every case can be traced back after the fact.

I hope some immutable store databases take off that cover those use cases. Make it a graph database, bonus points if the underlying engine implements directed hypergraph. And artificial intelligence reasoning engine. :)