Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The trouble with Cassandra as an object storage metadata database

blog.min.io

Post reply on HN
Print view View original

61–70 of 82 posts

Re: The trouble with Cassandra as an object storage metadata database

#61

Post by chrislusf »

I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users.

The article listed many known Cassandra characteristics and cited them as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are not convincing either. They are related to how to design the data structure better.

Actually, SeaweedFS can use many other database/KV stores as the metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores

I did find one drawback for Cassandra as the metadata store though. One use case is that the customer uploaded a lot of zip files to one folder /tmp, unzip them, and then moved to a final folder. The rate is about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow.

The solution was to use Redis for that /tmp folder, and still use Cassandra for the rest of folders. With Redis B-tree structure, the creation and deletion are cheap.

So it is all depends on use cases.

Re: The trouble with Cassandra as an object storage metadata database

#62

Post by adev_ »

ddorian43 wrote: post #16
adev_ wrote: post #8

Earlier quoted context omitted.

> Most of the time postgres will do fine. For something that scale horizontally, I would probably recommend to use something like FoundationDB for this use case [^1]. A transactional Key-Value store is exactly what you want for this kind of use case. [^1]: https://apple.github.io/foundationdb/

Yes now you are maintaining 5+ types of services for foundationdb. While you can vertically-scale a single PostgreSQL (with failover/ha) server to 50+TB of NVME and grow until you can throw other engineers at the problem. Example: wasabi.com uses mysql for metadata.

I'm curious on you describing the 5+ types of services fdb requires. In my world it requires 1.

Re: The trouble with Cassandra as an object storage metadata database

#63

Post by aasasd »

jeffbee wrote: post #40
manigandham wrote: post #39

Earlier quoted context omitted.

The issue seems to be a misunderstanding of what Cassandra is. It's an (advanced/nested) key-value database, so of course you need to specify the key to do anything.

The misunderstanding is understandable, since the Apache Cassandra site fails to state what Cassandra is, even directly below the heading "What is Cassandra?"

That's normal by now. Pretty much all software/service sites would do better by just using Wikipedia's description of themselves as the landing page. At least, better for the user. Probably because Wikipedia actually needs to explain the thing's meaning, and a standard marketing page would be slapped with plaques about non-encyclopedic content, in no time.

Re: The trouble with Cassandra as an object storage metadata database

#64

Post by snapetom »

SMFloris wrote: post #2

A while back we explored the use of Cassandra. We wanted to keep some event related data there and for it to be relatively fast read-wise in order for us to do all sorts of reporting based on it. So we wrote allot and wanted to read fast. Seemed like a perfect store for our timestamped events, especially since we wanted to not even use deletes and has in-build record deduplication via its primary key. Turns out, it i…

We initially looked at Cassandra. We liked its use case, we liked its scalability. However we also ran into maintenance and setup pains.

We ended up going with ScyllaDB, which is a drop in replacement for Cassandra. It’s written in C. Much easier in resource demands and we didn’t have to deal with Zookeeper directly.

Re: The trouble with Cassandra as an object storage metadata database

#65

Post by AtlasBarfed »

"I wanted to store something that could scale, but wanted to store and access my data in a way that didn't scale. Also, I don't understand CAP, distributed transactions, or distributed systems. I had a bad time."

Somewhere else in the comments: "Yeah, we went with MongoDB."

Re: The trouble with Cassandra as an object storage metadata database

#66

Post by bbromhead »

snapetom wrote: post #64
SMFloris wrote: post #2

A while back we explored the use of Cassandra. We wanted to keep some event related data there and for it to be relatively fast read-wise in order for us to do all sorts of reporting based on it. So we wrote allot and wanted to read fast. Seemed like a perfect store for our timestamped events, especially since we wanted to not even use deletes and has in-build record deduplication via its primary key. Turns out, it i…

We initially looked at Cassandra. We liked its use case, we liked its scalability. However we also ran into maintenance and setup pains. We ended up going with ScyllaDB, which is a drop in replacement for Cassandra. It’s written in C. Much easier in resource demands and we didn’t have to deal with Zookeeper directly.

Cassandra doesn't require Zookeeper

Re: The trouble with Cassandra as an object storage metadata database

#67

Post by garyrichardson »

jimbokun wrote: post #24
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

Best of summary of the Cassandra data model I've seen: HashMap > So you can lookup data by the ParititionKey, then perform range queries on the ClusterKey to filter data belonging to that partition. If your access patterns look like that, Cassandra could be a great fit. For anything else, you probably want to consider a different data store.

Making CQL SQL-Like was the worst possible thing the designers could have done. Newbies want to do SQL joins and filtering with it until they've been a few times.

C* is essentially scaled/automated MySQL Sharding and blob storage.. except you pay a huge cost for any server side filtering.

The Partition Key would be the equivalent of your MySQL shard id. ClusterKey is your primary key. if you treat everything like PUT $paritionKey, $primaryKey, $data and GET $partitionKey, $primaryKey you get a slow and massively scaling redis. If you do anything else with it you'll likely start regretting your choices and looking for a replacement database.

Re: The trouble with Cassandra as an object storage metadata database

#68

Post by _benedict »

aasasd wrote: post #43
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

Noob question here. > Your partitions need to be Why is that? Is that some kind of engineered-in limitation?

No, it's just that the storage layer is low priority to improve for this use case, but it'll happen eventually.

Re: The trouble with Cassandra as an object storage metadata database

#69

Post by cnlwsu »

noncoml wrote: post #60
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

> Your partitions need to be Is that a typo? 300MB sounds ridiculously low..

They don’t need to be. It’s just less performant so easier to just say that so people’s expectations on tps etc match. I’ve seen it in many GBs run fine

Re: The trouble with Cassandra as an object storage metadata database

#70

Post by cnlwsu »

aasasd wrote: post #43
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

Noob question here. > Your partitions need to be Why is that? Is that some kind of engineered-in limitation?

It’s not a real requirement. Just a performance thing. Each partition has an index and as it grows the deserialization of it to find the queried start column takes longer. It is lazy loads but there a perf hit. Easier to have a rule of thumb like “100mb partitions” and have more fixed tps per core etc expectations.
Post reply on HN