Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

infoq.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#11

Post by TideAd »

t90fan wrote: post #9
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very no…

Scaling up also takes up a lot of resources so you're never able to scale up in response to load without hosing your database even more.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#12

Post by pbreit »

hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#13

Post by avereveard »

pbreit wrote: post #12
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either.

There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments.

That is not something you'd be able to put on one single database in 2004.

Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that age looked at the numbers and went yeah not in a database.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#14

Post by 1letterunixname »

Yelp: the mafia of restaurant protection rackets. I don't care if they released a free quantum computer or solved world hunger because they're still making a living off harming small businesses.

https://www.yahoo.com/lifestyle/can-you-trust-yelp-crowd-fun...

https://thehustle.co/botto-bistro-1-star-yelp/

https://nypost.com/2014/10/13/restaurant-fights-yelps-allege...

https://www.wired.com/2010/02/yelp-sued-for-alleged-extortio...

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#15

Post by onlypositive »

avereveard wrote: post #13
pbreit wrote: post #12

Earlier quoted context omitted.

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

Cassandra was initially released in 2008.

Back in 2004 I was running larger MySQL databases than yelp has now:

https://www.enterpriseappstoday.com/stats/yelp-statistics.ht...

> That is not something you'd be able to put on one single database in 2004.

Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication.

But even if it wasn't: just don't run it all on one db? Nothing says you need to be able to join on the restaurant table and the comments table. Put them on different servers. That's all you're doing with Casandra anyway.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#16

Post by weego »

avereveard wrote: post #13
pbreit wrote: post #12

Earlier quoted context omitted.

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

I mean, for a start Cassandra wasn't available for years after that date, but either way, whatever Postgres was like back then Cassandra was much more of a piece of shit. Smells like implementing what the cool kids were doing to me.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#17

Post by hardwaresofton »

t90fan wrote: post #9
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very no…

Thanks for sharing your experience -- I know I've spent a lot of time in the past worrying about FS corruption, but generally expecting that the database sitting on top of it should never get corrupted, mostly because I use postgres so much.

I don't have the experience you do in this situation, but my first reaction to this was definitely "don't use Cassandra". But I also never really understood the use-case where Cassandra shines as a solution either (seems like only companies with a lot of data really seem to get wins from it?)

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#18

Post by darkstar_16 »

t90fan wrote: post #9
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very no…

We run a Cassandra cluster in production and its a pretty small cluster yet all that you mentioned seems to resonate. We do use Cassandra reaper to automate some of the tasks but no one wants to touch Cassandra in general in the team.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#19

Post by tomnipotent »

onlypositive wrote: post #15
avereveard wrote: post #13

Earlier quoted context omitted.

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on o…

Postgres didn't ship built-in replication until 2010, and prior replication solutions like Slony were not options I would have enjoyed building a business around.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#20

Post by redeyedtreefrog »

pbreit wrote: post #12
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

I found this on their tech blog from 2016: https://engineeringblog.yelp.com/2016/08/how-we-scaled-our-a...

Whether they really needed cassandra or whether it was just using fancy complicated tech for the sake of it, I couldn't say.

Post reply on HN