Live data from Hacker News

Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

infoq.com

41–50 of 53 posts

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#41
post #33

Earlier quoted context omitted.

There is always a better solution than Cassandra, until your data will no longer fit on a single server or you actually need guaranteed availability.

Is there no other db that offers guaranteed availability? I remember reading about Discord switching from Cassandra to ScyllaDB I think.

ScyllaDB Is just knock off cassandra with different features and performance characteristics.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#42
post #12

Earlier quoted context omitted.

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

About 6 years ago I was brought in as a Azure/operation consultant to help bid on a "project". Basically a sister-company had built a webapp for a client and we where to bid on hosting and migration away from the current fly-by-night hosting partner. The specs called for a Kubernetes cluster and a Cassandra database cluster for production and the same for testing. Everything about this project was presented as being…

Hosted k8s to run a few docker image is almost no extra work and possibly less work than setting up your own docker hosting system.

I see a small hosted k8s setup with a hosted sql server as a fine app for many things.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#43

Earlier quoted context omitted.

I found this on their tech blog from 2016: https://engineeringblog.yelp.com/2016/08/how-we-scaled-our-a... Whether they really needed cassandra or whether it was just using fancy complicated tech for the sake of it, I couldn't say.

Yeah sounds like they liked it because of the infinitely scalable storage and high write throughput, but don’t really get into if they were having issues with all that on MySQL. Also, they use the term “analytics” a lot, but it seems to be a different use case than typical OLAP workloads since they were serving this data to customers adhoc.

Scaling writes on MySQL/Postgres is a huge PITA once you hit the point where a single master on a chonky server can't handle it anymore and you lose most of the benefits of being on a relational db.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#44

Earlier quoted context omitted.

Transaction in this case is not a database transaction, but a financial transaction (payment). Per payment, probably somewhere in the order of 50-100 database transactions (although Cassandra does not really have transactions of course, interpret this as read/write actions) will be performed in the course of its processing. So that is 1,875,000,000,000 database actions on busy days. Not a DBA, but for our purposes th…

There are three extra zeros here

Whoops you are right.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#45

Earlier quoted context omitted.

About 6 years ago I was brought in as a Azure/operation consultant to help bid on a "project". Basically a sister-company had built a webapp for a client and we where to bid on hosting and migration away from the current fly-by-night hosting partner. The specs called for a Kubernetes cluster and a Cassandra database cluster for production and the same for testing. Everything about this project was presented as being…

Hosted k8s to run a few docker image is almost no extra work and possibly less work than setting up your own docker hosting system. I see a small hosted k8s setup with a hosted sql server as a fine app for many things.

True, but it's a complete waste if all you need is a single webapp and you're running it on something like Azure anyway which can just run that single Docker image.

In this case you had a potential zero management environment, truly Cloud as it was meant to be, vs. managing Kubernetes, plus a database. You could go with managed Kubernetes (AKS) and a managed SQLServer, but why take on that cost?

Edit: Even AKS isn't truly zero management, you need to do at least some of the work for the upgrades, so instantly more management, something you need to do, something that adds to the operational cost.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#46

Earlier quoted context omitted.

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

Cassandra was initially released in 2008. Back in 2004 I was running larger MySQL databases than yelp has now: https://www.enterpriseappstoday.com/stats/yelp-statistics.ht... > That is not something you'd be able to put on one single database in 2004. Sorry, but it was, plenty of companies had much larger databases back then running plain old master slave replication. But even if it wasn't: just don't run it all on o…

If you're not just running it all on one db, why do you have to have all of your different DBs be on the same DB software?

I used to work at Yelp, and at least at that time, a big use case of Cassandra was basically for what were essentially materialized views created from log data. At that point in time, MySQL or logs were the "sources of truth", but there were enough transactions going on that it made sense to have things like Cassandra around too for some of the other use cases.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#47
post #12

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Yelp's main database was a couple of MySQL clusters (several TB of total data) for almost all of the company's existence, but they've always had a penchant for architecture astronomy. The ad system had the most justification for novel data architectures historically (when I was there, it was generating and processing tens of terabytes of data per day via Hadoop/EMR, which was new and cool, and I'm sure it's at least an order of magnitude mode than that now).

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#48

Yelp: the mafia of restaurant protection rackets. I don't care if they released a free quantum computer or solved world hunger because they're still making a living off harming small businesses. https://www.yahoo.com/lifestyle/can-you-trust-yelp-crowd-fun... https://thehustle.co/botto-bistro-1-star-yelp/ https://nypost.com/2014/10/13/restaurant-fights-yelps-allege... https://www.wired.com/2010/02/yelp-sued-for-allege…

Not just restaurants. They attempted to extort my farm after I deleted my account. I kinda wish that the cluster had been lost; that would have been the most beneficial outcome for society.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#49
post #33

Earlier quoted context omitted.

There is always a better solution than Cassandra, until your data will no longer fit on a single server or you actually need guaranteed availability.

Is there no other db that offers guaranteed availability? I remember reading about Discord switching from Cassandra to ScyllaDB I think.

Yes, other distributed databases like MongoDB, CockroachDB, probably a few others. Or even multi-master DB setups. As with Cassandra, you don't want to use them unless you really need that availability and can suffer the downsides. It seems pretty rare to actually need those availability guarantees, rather than say a robust fast failover setup which might cancel some in flight transactions. It is probably when you start looking at two phase commit that you look for alternatives with better availability stories.
Post reply on HN