Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture • Hacker News Board

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#21

Post by CHY872 » Thu, Jul 20, 2023, 8:33 AM UTC

hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

I have >5 years of experience using Cassandra in production, involving thousands of clusters storing petabytes of data. My conclusion from that time is that Cassandra is simply not robust enough to be a general purpose database (the team are working on it but they're coming from a really rough starting place) - there are lots of ways to cause data corruption, and Cassandra does enough dynamic repairing that it can be hard to catch this before your backups are dropped due to time windowing. Unfortunately, the juice may still be worth the squeeze - Cassandra's storage model lends itself very nicely to disaster recovery workflows in a way which something like Oracle or FoundationDB does not (and it's Cassandra so you'll need it!), while the ability to horizontally scale gets you out of so many operations issues. If you've got a schema which works well in Cassandra, you've probably solved a lot of the issues you might have.

Example of fairly standard Cassandra bug (don't know if present on latest release, certainly was a year or two ago): When you add a new node to the cluster, it 'bootstraps', where it copies ~1/n the data from other nodes. When you are done bootstrapping, it's copied a bunch of data from other nodes, but the other nodes still contain that data. You then run 'cleanups' on the other nodes to remove the (now stale and unusable) data so as to get your disk space back.

If you accidentally run a cleanup on the new node as it is being bootstrapped, it will succeed, you will delete all the data that's been copied over so far, and Cassandra will _not_ terminate the bootstrap. Everything will be green, but your new node will suddenly be using 0 disk space. When the bootstrap finishes, possibly days later, your cluster will be immediately corrupted due to violated replication guarantees - but only on data that hasn't been read or written over that period, because if it was written it'll be re-replicated, and if it was read Cassandra will silently repair at this time. Repairs resolve the issue, but if you've made this mistake due to scripting, if you get unlucky it's possible to just delete all replicas of some data between repairs.

Example of other Cassandra bug (again, might be outdated): Cassandra nodes identify themselves on startups with IPs, and the owned token ranges are not persisted, they're streamed from other nodes in the cluster. If you've deployed your Cassandra in K8s and you reboot multiple nodes in one go and they swap IPs upon reboot, you may now find yourself in a split brain situation in which nodes magically forget they own certain data ranges and think they own each others data (or maybe it's that the nodes still think they own the right ranges but other nodes think they own the wrong ranges). Wasn't close enough to fully debug that one.

It's a mess. Would seek to avoid problem spaces where I might need to use it again, though if by chance ended up in a space where it made sense, probably wouldn't avoid the tech.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#22

Post by beoberha » Thu, Jul 20, 2023, 8:35 AM UTC

redeyedtreefrog wrote: post #20

pbreit wrote: post #12

Earlier quoted context omitted.

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

I found this on their tech blog from 2016: https://engineeringblog.yelp.com/2016/08/how-we-scaled-our-a... Whether they really needed cassandra or whether it was just using fancy complicated tech for the sake of it, I couldn't say.

Yeah sounds like they liked it because of the infinitely scalable storage and high write throughput, but don’t really get into if they were having issues with all that on MySQL. Also, they use the term “analytics” a lot, but it seems to be a different use case than typical OLAP workloads since they were serving this data to customers adhoc.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#23

Post by mrweasel » Thu, Jul 20, 2023, 8:59 AM UTC

pbreit wrote: post #12

hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

About 6 years ago I was brought in as a Azure/operation consultant to help bid on a "project". Basically a sister-company had built a webapp for a client and we where to bid on hosting and migration away from the current fly-by-night hosting partner.

The specs called for a Kubernetes cluster and a Cassandra database cluster for production and the same for testing. Everything about this project was presented as being highly complex. Digging into it with the client and the developers we reduced it to: One Docker container running in Azure websites and CosmosDB on the backend, which could basically run on the free tier.

The whole thing hold less than 3GB of data and had maybe a few hundred requests per day. The client just loved the idea that they where special, dealing with massive amount of data and required scalability to keep costs under control. The developers more or less just ran with it and wanted to do Kubernetes and Cassandra sounded interesting and now they had a client that would pay for it. Technically I suppose that both Kubernetes and Cassandra where reasonable choices, had they had 1000x the load, but given their market they where never going to grow beyond 10x on this particular solution.

We didn't get the contract. Our bid was insanely low (not really worth the cost of bringing in a new contractor), delivered something different that asked for (fair enough).

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#24

Post by Kototama » Thu, Jul 20, 2023, 9:02 AM UTC

onlypositive wrote: post #15

avereveard wrote: post #13

Earlier quoted context omitted.

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on o…

Cassandra is highly-available, your suggestion is not.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#25

Post by leokennis » Thu, Jul 20, 2023, 9:15 AM UTC

t90fan wrote: post #9

hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very no…

We (as in, my company, not me myself) run large Cassandra clusters in the critical path of bank transaction processing (in the order of 2-25 million payments per day, each requiring a lot of database queries) and it's going pretty well...

https://www.youtube.com/watch?v=0QsLU9na2uE

But yes, you win some (mainly resilience, availability and disaster avoidance, possibly tunable consistency will help you) you lose some.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#26

Post by hsjqllzlfkf » Thu, Jul 20, 2023, 9:24 AM UTC

leokennis wrote: post #25

t90fan wrote: post #9

Earlier quoted context omitted.

Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very no…

We (as in, my company, not me myself) run large Cassandra clusters in the critical path of bank transaction processing (in the order of 2-25 million payments per day, each requiring a lot of database queries) and it's going pretty well... https://www.youtube.com/watch?v=0QsLU9na2uE But yes, you win some (mainly resilience, availability and disaster avoidance, possibly tunable consistency will help you) you lose some.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#27

Post by onlypositive » Thu, Jul 20, 2023, 9:26 AM UTC

Kototama wrote: post #24

onlypositive wrote: post #15

Earlier quoted context omitted.

Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on o…

Cassandra is highly-available, your suggestion is not.

MySQL master slave replication has failover, so it is technically highly available, especially if you have multiple slaves. It was also best practice back then so this argument is kind of weird.

Also, whether you need a highly available database for a company like yelp is an implementation detail.

Oh no, the database is down, how will Jonny decide where to go for lunch now?!

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#28

Post by onlypositive » Thu, Jul 20, 2023, 9:29 AM UTC

tomnipotent wrote: post #19

onlypositive wrote: post #15

Earlier quoted context omitted.

Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on o…

Postgres didn't ship built-in replication until 2010, and prior replication solutions like Slony were not options I would have enjoyed building a business around.

MySQL shipped replication in 2000. This whole conversation is some webscale copium bullshit. I'm out.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#29

Post by Kototama » Thu, Jul 20, 2023, 9:53 AM UTC

onlypositive wrote: post #27

Kototama wrote: post #24

Earlier quoted context omitted.

Cassandra is highly-available, your suggestion is not.

MySQL master slave replication has failover, so it is technically highly available, especially if you have multiple slaves. It was also best practice back then so this argument is kind of weird. Also, whether you need a highly available database for a company like yelp is an implementation detail. Oh no, the database is down, how will Jonny decide where to go for lunch now?!

Cassandra is more close to AP whereas MySQL with clustering is more like CP. So they serve different purpose. See this nice discussion on Stackoverflow: https://stackoverflow.com/questions/36404765/why-isnt-rdbms-...

> Also, whether you need a highly available database for a company like yelp is an implementation detail.

You surely know better than them what they need, that's impressive.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#30

Post by [deleted] » Thu, Jul 20, 2023, 10:55 AM UTC

avereveard wrote: post #13

pbreit wrote: post #12

Earlier quoted context omitted.

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

[deleted]