The trouble with Cassandra as an object storage metadata database

We use it in anger at Splitgraph [0] to let people upload/download datasets and Postgres table snapshots (instead of storing them directly in S3).

Pros:

* Less platform-dependent. By self-managing it, we can also run deploys to GCP / Azure / Scaleway / other providers without writing a separate adapter for e.g. Azure Blob Storage.

* Python API [1] much more pleasant to use than boto3 (and can speak to normal S3). It doesn't do everything that boto3 does, but it supports everything we need (e.g. pre-signed URLs).

* minio server itself supports a large chunk of S3's functionality (e.g. SELECT API / AssumeRole / bucket versioning)

* Don't pay per request and for egress: this was a big deal since people might want to download large amounts of data from us (or make a bunch of small requests to download/upload a subset of data).

Cons:

* Have to manage own infrastructure. We run it on managed VMs so it's semi-managed, but we still have to provision block storage, set up backup policies etc.

* In a similar vein, scaling and availability all have to be DIY [2]. We haven't run into situations yet where Minio would be the bottleneck, but it might be something to keep in mind.

* Obviously not as seamless: you don't get things like Glacier or integration with other IAM.

[0] https://www.splitgraph.com/

[1] https://github.com/minio/minio-py

[2] https://docs.min.io/docs/distributed-minio-quickstart-guide....

> What are the pros & cons over vanilla s3

Mostly that you aren't bound to AWS IMHO. You can run it on-prem, or in a cloud provider with no object storage service.

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional.

Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition key column is "day_of_week", then you have 7 partitions. New problem: Your partitions need to be In fact you'll soon realize that over the lifetime of your table, keeping those partitions under control may end up forcing you to use various date tricks like putting year/month/day in as "artificial" partition keys.

Of course if you put everything in the partition keys instead of clustering keys, let me note again that you have to put each partition key column in every where clause, in which case you may have trouble querying for batches of data.

Furthermore, when you do put clustering columns in your where clause, they have to be in order declared; so if your CC's are a, b, and c, then your where clause can use (a), (a,b), or (a,b,c); but if it has b, it has to have a, and if it has c, it has to have a&b. This is because storage is hierarchical. (Same rules for "order by", btw) (and no there's no "group by")

This is when you start realizing: Oh, you mean it's really not like "SQL without joins". No, not even close.

It doesn't work for a lot of small files scenario, there is too much overhead. You can't +1 new machines, you need to add whole clusters. If you ask too many questions you get invited for a premium subscription. You can't add +1 server, you need to add in clusters. There is no public big production scenario of 10PB+. There is no community of big production users (like there is with ceph).