Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The trouble with Cassandra as an object storage metadata database

blog.min.io

Post reply on HN
Print view View original

71–80 of 82 posts

Re: The trouble with Cassandra as an object storage metadata database

#71

Post by AtlasBarfed »

MichaelMoser123 wrote: post #31
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

> New problem: Your partitions need to be people work around that by storing the data in something like S3 and then keeping the handle in cassandra. Yet another hack on top of another hack.

Do you fill postgres with 100MB+ blobs?

You'd do the same "hack" there.

Re: The trouble with Cassandra as an object storage metadata database

#72

Post by AtlasBarfed »

killtimeatwork wrote: post #7
SMFloris wrote: post #2

A while back we explored the use of Cassandra. We wanted to keep some event related data there and for it to be relatively fast read-wise in order for us to do all sorts of reporting based on it. So we wrote allot and wanted to read fast. Seemed like a perfect store for our timestamped events, especially since we wanted to not even use deletes and has in-build record deduplication via its primary key. Turns out, it i…

I think Cassandra is a better fit for interactive use cases, not for reporting. Also, basically it's super heavy duty and it should start to shine when you're really serving entire Internet (on the scale of Reddit, Expedia etc.) and your Cassandra cluster is distributed across DCs across the world. I haven't really worked in this space for a couple of years so I don't know if the cloud offerings have already complete…

Dynamo allegedly has global replication now.

Of course, I have been totally unable to determine how they merge rows/partitions without cell timestamps. It's a black box.

I was just on a "Keyspaces" meeting where the sales dude basically described dynamo billing, dynamo provisioning, feature shortfalls obviously due to dynamo, but would not admit it was dynamo.

It was bizarre.

Re: The trouble with Cassandra as an object storage metadata database

#73

Post by AtlasBarfed »

bbromhead wrote: post #66
snapetom wrote: post #64

Earlier quoted context omitted.

We initially looked at Cassandra. We liked its use case, we liked its scalability. However we also ran into maintenance and setup pains. We ended up going with ScyllaDB, which is a drop in replacement for Cassandra. It’s written in C. Much easier in resource demands and we didn’t have to deal with Zookeeper directly.

Cassandra doesn't require Zookeeper

And Scylladb may have a better thread-per-core model and no GC pauses, it basically has the exact same management challenges as Cassandra.

The parent comment almost seems generated by AI.

Re: The trouble with Cassandra as an object storage metadata database

#74

Post by ddorian43 »

adev_ wrote: post #62
ddorian43 wrote: post #16

Earlier quoted context omitted.

Yes now you are maintaining 5+ types of services for foundationdb. While you can vertically-scale a single PostgreSQL (with failover/ha) server to 50+TB of NVME and grow until you can throw other engineers at the problem. Example: wasabi.com uses mysql for metadata.

I'm curious on you describing the 5+ types of services fdb requires. In my world it requires 1.

Heading to the docs again, looks like they were consolidated. I remember you had to run coordinator nodes, data nodes, log storage nodes, cluster controller etc.

Re: The trouble with Cassandra as an object storage metadata database

#75

Post by ddorian43 »

noncoml wrote: post #60
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

> Your partitions need to be Is that a typo? 300MB sounds ridiculously low..

They have a bad data structure in it that it's so low.

Re: The trouble with Cassandra as an object storage metadata database

#76

Post by ddorian43 »

chrislusf wrote: post #61

I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users. The article listed many known Cassandra characteristics and cited them as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are no…

> With Redis B-tree structure, the creation and deletion are cheap.

Pretty sure redis has no b-tree but hashmap. But redis has to do no compaction at all since it can directly delete the stuff on disk.

> The rate is about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow.

The problem here also is the gc-perdiod. Since the database has async-replication, it needs to keep deleted rows for some days so downed replicas can replicate. So they stay in disk/memory for some days actually and don't get compacted. While in a CP system they would be.

Re: The trouble with Cassandra as an object storage metadata database

#77

Post by ddorian43 »

gryn wrote: post #49

can anyone explain to me why someone would want to use cassandra when not handling internet scale stuff? the team am in uses it, but after many times asking why it was chosen since it seems a poor fit for our uses cases compared to a relational DB, the only justification I was given is a that the cluster is easier to maintain for our ops guys.

Contrary to popular beliefs, programmers aren't very rational people, but still prone to emotional beliefs when choosing databases.

Another reason is for "resume driven development".

Re: The trouble with Cassandra as an object storage metadata database

#78

Post by ddorian43 »

mthomassen wrote: post #45
pid_0 wrote: post #23

From an ops perspective, managing Cassandra is a bitch. Just use a managed service unless you have the money to hire a dedicated Cassandra expert. I’m so glad I’m done with ops

Just to provide a counter statement; at $work we definatively do not have the appropriate headcount to even start considering running our own distrubuted datastores. However, we _do_ use Cassandra. And while many persons shared the sentiment of 'it being to hard to maintain' when the first few projects started to incorporate Cassandra, the truth is; there never has been any failure or incident, or even technical hurd…

Yes, but 200GB and those read/write rates of data can reside in a single server even entirely in-memory, it's not a big amount.

Re: The trouble with Cassandra as an object storage metadata database

#79

Post by MichaelMoser123 »

AtlasBarfed wrote: post #71
MichaelMoser123 wrote: post #31

Earlier quoted context omitted.

> New problem: Your partitions need to be people work around that by storing the data in something like S3 and then keeping the handle in cassandra. Yet another hack on top of another hack.

Do you fill postgres with 100MB+ blobs? You'd do the same "hack" there.

yes, but postgress doesn't have the problem that it fails if you have more than 100MB in all records, in total.

Re: The trouble with Cassandra as an object storage metadata database

#80

Post by snapetom »

AtlasBarfed wrote: post #73
bbromhead wrote: post #66

Earlier quoted context omitted.

Cassandra doesn't require Zookeeper

And Scylladb may have a better thread-per-core model and no GC pauses, it basically has the exact same management challenges as Cassandra. The parent comment almost seems generated by AI.

Dumb statements all around, but clearly you've never been in a Cassandra environment vs Scylla. Scylla is far, far more reliable, easier to get up and running, and required a bit less supervision than Cassandra.
Post reply on HN