Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The trouble with Cassandra as an object storage metadata database

blog.min.io

Post reply on HN
Print view View original

31–40 of 82 posts

Re: The trouble with Cassandra as an object storage metadata database

#31

Post by MichaelMoser123 »

kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

> New problem: Your partitions need to be people work around that by storing the data in something like S3 and then keeping the handle in cassandra. Yet another hack on top of another hack.

Re: The trouble with Cassandra as an object storage metadata database

#32

Post by CraigJPerry »

mrwd021 wrote: post #30
segmondy wrote: post #9

If you really want to see and get hints on what you can do with Cassandra, head over to Netflix tech blogs. They use Casandra extensively.

really? what do they use Cassandra for?

Back in 2014 when i last used Cassandra, Netflix were using it for some kind of metrics store i believe. They also published the nicer-to-use client back then (astyanx or something it was called - nicer API than Hector).

My memory is letting me down here on specifics - in my defence it's been a long time since i answered questions on stack overflow about Cassandra. I absolutely loved working with the product even if the sharp edges cut me a few times (ultimately my own fault not Cassandra's).

Re: The trouble with Cassandra as an object storage metadata database

#33

Post by jeffbee »

uDontKnowMe wrote: post #29
KaiserPro wrote: post #6

I would have hoped its obvious. Your metadata database needs to be the fastest and most reliable store out of everything. It can't be eventually consistent without partitioning your datastore. Even then you'll end up partitioning your data neatly into the same failure zone. Cassandra has basically one usecase: high volume writes, with a few batch reads. Cassandra is not really optimised for high reads. Most of the ti…

What would you use instead of Cassandra for multi-DC replication?

Cloud bigtable.

Re: The trouble with Cassandra as an object storage metadata database

#34

Post by anonymousDan »

ddorian43 wrote: post #14

Storing metadata together with data will just make it harder,slower to query metadata (since it will reside in hdd in most cases). You may think "it will be cached in ram, because it's small", yes, but then you'll end up querying many nodes just for metadata queries. Yes it's nicer to manage only 1 system, but in big scenarios it's probably better to separate. You can have 50+TB of nvme in 1 server, so your metadata…

What kind of 'metadata' are they referring to here? Can you give some concrete examples?

Re: The trouble with Cassandra as an object storage metadata database

#35

Post by anonymousDan »

I don't quite get what they are referring to when they talk about 'metadata' here. Are they talking mostly about something internal to the database, something to do with the schema, or some kind of additional data used to enrich a particular object?

Re: The trouble with Cassandra as an object storage metadata database

#36

Post by redis_mlc »

mrwd021 wrote: post #30
segmondy wrote: post #9

If you really want to see and get hints on what you can do with Cassandra, head over to Netflix tech blogs. They use Casandra extensively.

really? what do they use Cassandra for?

Netflix uses it for everything that you'd use Postgres for but:

- they have Cassandra committers (most end up going to Apple iCloud)

- they have mgmt. support to make it work

- they are/were on tokens for years, not vtokens, since their internal tools were token-based

(They started with Oracle Enterprise (and a little MySQL), tried Mongo briefly, then decided to make Cassandra work for them.)

Source: original Netflix Cassandra team member.

Re: The trouble with Cassandra as an object storage metadata database

#37

Post by ddorian43 »

anonymousDan wrote: post #34
ddorian43 wrote: post #14

Storing metadata together with data will just make it harder,slower to query metadata (since it will reside in hdd in most cases). You may think "it will be cached in ram, because it's small", yes, but then you'll end up querying many nodes just for metadata queries. Yes it's nicer to manage only 1 system, but in big scenarios it's probably better to separate. You can have 50+TB of nvme in 1 server, so your metadata…

What kind of 'metadata' are they referring to here? Can you give some concrete examples?

Metadata of the object storage. In the simplest is a table of

Re: The trouble with Cassandra as an object storage metadata database

#38

Post by [deleted] »

jimbokun wrote: post #24
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

Best of summary of the Cassandra data model I've seen: HashMap > So you can lookup data by the ParititionKey, then perform range queries on the ClusterKey to filter data belonging to that partition. If your access patterns look like that, Cassandra could be a great fit. For anything else, you probably want to consider a different data store.

[deleted]

Re: The trouble with Cassandra as an object storage metadata database

#39

Post by manigandham »

kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

The issue seems to be a misunderstanding of what Cassandra is. It's an (advanced/nested) key-value database, so of course you need to specify the key to do anything.

Re: The trouble with Cassandra as an object storage metadata database

#40

Post by jeffbee »

manigandham wrote: post #39
kerblang wrote: post #15

The most basic and trivial way that you're going to get burned by cassandra is that you have to divide your primary key into two parts: partition key columns, and clustering key columns. Partition keys must be in every "where" clause; only clustering keys are optional. Okay, I'll just not bother with partition keys, right? Except partition keys determine which "partition" your data goes in. So if your only partition…

The issue seems to be a misunderstanding of what Cassandra is. It's an (advanced/nested) key-value database, so of course you need to specify the key to do anything.

The misunderstanding is understandable, since the Apache Cassandra site fails to state what Cassandra is, even directly below the heading "What is Cassandra?"
Post reply on HN