Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The trouble with Cassandra as an object storage metadata database

blog.min.io

Post reply on HN
Print view View original

41–50 of 82 posts

Re: The trouble with Cassandra as an object storage metadata database

#41

Post by mrweasel »

francoisdevlin wrote: post #5

A tangent - I love using minio in dev & test for their s3 simulators. Being able to throw away a bucket and start from scratch, and having everything self contained in my docker-compose command is a real blessing. Has anyone ever used min.io for production stuff? What are the pros & cons over vanilla s3?

We opted against Minio, mostly because we had a storage solution that provide S3 api, but also because managing Minio seemed painful. It’s just not clearly versioned, we didn’t figure out how you where suppose to rotate cerificates and documentation isn’t great.

Re: The trouble with Cassandra as an object storage metadata database

#42

Post by acscott »

omginternets wrote: post #28

This is a restatement of the Cassandra docs paired with the usual misunderstanding of CAP. An AP system is not a choice of "I'll have availability and partition tolerance, please". It's the choice of availability given a partition. The whole point of CAP is that when partitions occur, there is a forced choice -- this is why it's incorrect to ask for a CA system. The Minio team are manifestly excellent engineers, but…

Do you mean AP is the choice of losing consistency given a partition?

Re: The trouble with Cassandra as an object storage metadata database

#43

Post by aasasd »

kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

Noob question here.

> Your partitions need to be

Why is that? Is that some kind of engineered-in limitation?

Re: The trouble with Cassandra as an object storage metadata database

#44

Post by aasasd »

MichaelMoser123 wrote: post #31
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

> New problem: Your partitions need to be people work around that by storing the data in something like S3 and then keeping the handle in cassandra. Yet another hack on top of another hack.

Pretty sure that's what the article is about.

Re: The trouble with Cassandra as an object storage metadata database

#45

Post by mthomassen »

pid_0 wrote: post #23

From an ops perspective, managing Cassandra is a bitch. Just use a managed service unless you have the money to hire a dedicated Cassandra expert. I’m so glad I’m done with ops

Just to provide a counter statement; at $work we definatively do not have the appropriate headcount to even start considering running our own distrubuted datastores. However, we _do_ use Cassandra. And while many persons shared the sentiment of 'it being to hard to maintain' when the first few projects started to incorporate Cassandra, the truth is; there never has been any failure or incident, or even technical hurdle related to the usage/maintenance of the -admittedly smallish- Cassandra cluster (for 2 years now).

The costs of operating the cluster are ~5K/month (that's what our service provider charges us for 24/7 ops). I consider this a scam since averaged maintenance costs are perhaps ~1 hour per month.

(normally peaks at 40.000 writes/sec, 500 reads/sec, 200GB)

Re: The trouble with Cassandra as an object storage metadata database

#46

Post by manigandham »

aasasd wrote: post #43
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

Noob question here. > Your partitions need to be Why is that? Is that some kind of engineered-in limitation?

There's a default limit of 2 billion cells (rows * cols) but larger partitions also mean larger indexes and more serialization, repair, compaction and memory overhead which doesn't work well with Cassandra's Java/JVM GC requirements.

It's getting better but likely will never reach the performance potential of Scylla which is Cassandra reimplemented in C++ with a much better sharded architecture.

Re: The trouble with Cassandra as an object storage metadata database

#47

Post by nemothekid »

anonymousDan wrote: post #35

I don't quite get what they are referring to when they talk about 'metadata' here. Are they talking mostly about something internal to the database, something to do with the schema, or some kind of additional data used to enrich a particular object?

Minio is an s3 clone, so I assume you metadata they mean details about the object you uploaded. So if the actual data is a video file, the metadata would be the headers, filesize, what host/drive the file is stored on. Essentially when you ask for minio.io/my/video/file some service has to transform that to 192.168.0.1/customer/drive/ahe123.blob

Re: The trouble with Cassandra as an object storage metadata database

#48

Post by Uberphallus »

KaiserPro wrote: post #6

I would have hoped its obvious. Your metadata database needs to be the fastest and most reliable store out of everything. It can't be eventually consistent without partitioning your datastore. Even then you'll end up partitioning your data neatly into the same failure zone. Cassandra has basically one usecase: high volume writes, with a few batch reads. Cassandra is not really optimised for high reads. Most of the ti…

> Cassandra has basically one usecase: high volume writes, with a few batch reads.

That's how we use it in BigCompany(tm).

Cassandra holds (almost) all of our data. Most of it comes from batch insertions, but more complex (and lower volume) data comes from various REST APIs we expose and other teams use.

Most reads (60%) done on it are boring batch reads: analytics, reports, deltas, etc.

For more complex reads (30%) we copy relevant rows to Hive tables, then perform queries there without dealing with partition key shenanigans. So, batch with extra steps.

Then anything that requires random reads (10%) is read from ElasticSearch:

1. Coming from REST backend if it's time sensitive.

2. Coming from a batch that reads from Cassandra if not.

Re: The trouble with Cassandra as an object storage metadata database

#49

Post by gryn »

can anyone explain to me why someone would want to use cassandra when not handling internet scale stuff?

the team am in uses it, but after many times asking why it was chosen since it seems a poor fit for our uses cases compared to a relational DB, the only justification I was given is a that the cluster is easier to maintain for our ops guys.

Re: The trouble with Cassandra as an object storage metadata database

#50

Post by [deleted] »

acscott wrote: post #42
omginternets wrote: post #28

This is a restatement of the Cassandra docs paired with the usual misunderstanding of CAP. An AP system is not a choice of "I'll have availability and partition tolerance, please". It's the choice of availability given a partition. The whole point of CAP is that when partitions occur, there is a forced choice -- this is why it's incorrect to ask for a CA system. The Minio team are manifestly excellent engineers, but…

Do you mean AP is the choice of losing consistency given a partition?

[deleted]
Post reply on HN