The trouble with Cassandra as an object storage metadata database

So these guys have a storage system on top of Amazon S3 with a distributed (timestamp based) RW mutex locking, that uses NTP (github.com/beevik/ntp). That seems to be about it.

https://github.com/minio/minio/blob/master/pkg/bucket/object...

https://github.com/minio/minio/blob/master/pkg/dsync/drwmute...

And this is their test:

https://github.com/minio/minio/blob/master/pkg/dsync/dsync_t...

I just browsed quickly but it is littered with Amazon Simple Storage Service hardcoded bits like this:

https://github.com/minio/minio/blob/master/pkg/bucket/object...

There is not a single document that I can find that discusses the MinIO architecture. I guess "MinIO is a simple wrapper around S3 with a homegrown distributed state tech using NTP and it is 'fast!'" does not make for a sexy doc.

The column "trouble with OSS distributed DBs" without Jepsen tests has probably already been written. There should also be one about "competitor's mature product bashing blogs are HN clickbait".

Best of summary of the Cassandra data model I've seen:

HashMap>

So you can lookup data by the ParititionKey, then perform range queries on the ClusterKey to filter data belonging to that partition.

If your access patterns look like that, Cassandra could be a great fit.

For anything else, you probably want to consider a different data store.

What you are expected to do with Cassandra, to achieve your horizontal scalability, is (a) denormalize by each supported query up front, and (b) fan out all of those (often by time) so they never get too big.

So for your "primary" data store for a type data, you might use the date for the partition key. Then if you do a lot of day-of-week queries, you'd store a second copy grouped by year-month-weekday, and query as many months as you need, and combine them. If you are collecting data for several sites, have a copy that's stored by year-month-weekday-site. And set up your software so it's easy to say "write in these 5 places, keyed off this combination of values." And because it's in all these denormalized places, you can forget updating anything atomically. So you go to append-only mode and if you have an "update" you record that as a separate entry after the first and your application's understanding of the data store includes coalescing the events and applying the update. (Maybe if you need it you have a separate process coalescing updates after the fact, but in the background, non-critical-like.)

And you're right. This is very much not SQL. It's also super obnoxious and support for it in extant frameworks is pretty minimal (they tend to be rather myopically SQL-oriented, ill suited to take advantage of this model). It also uses lots of storage, on the premise that disk is cheap. But what all this accomplishes is crazy horizontal scalability, and when you make any one query, then all the data is right there on disk lined up neatly, so it all gets streamed out in a predictable amount of time. For a few select applications, this consistent scalability is worth the pain.

This is a restatement of the Cassandra docs paired with the usual misunderstanding of CAP. An AP system is not a choice of "I'll have availability and partition tolerance, please". It's the choice of availability given a partition. The whole point of CAP is that when partitions occur, there is a forced choice -- this is why it's incorrect to ask for a CA system.

The Minio team are manifestly excellent engineers, but insightless posts that contain subtle misunderstandings of CAP do nothing to showcase that competence.

really? what do they use Cassandra for?