Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

infoq.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#31

Post by hobs »

hsjqllzlfkf wrote: post #26
leokennis wrote: post #25

Earlier quoted context omitted.

We (as in, my company, not me myself) run large Cassandra clusters in the critical path of bank transaction processing (in the order of 2-25 million payments per day, each requiring a lot of database queries) and it's going pretty well... https://www.youtube.com/watch?v=0QsLU9na2uE But yes, you win some (mainly resilience, availability and disaster avoidance, possibly tunable consistency will help you) you lose some.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

No need to add the extra bit - at the top end 289 transactions per seconds is not something you'd probably want to choose SQLite for, but PG/MySQL/SQL Server would do that fine and require a lot less feeding (though any database with traffic or size needs some care.)

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#32

Post by hobs »

weego wrote: post #16
avereveard wrote: post #13

Earlier quoted context omitted.

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

I mean, for a start Cassandra wasn't available for years after that date, but either way, whatever Postgres was like back then Cassandra was much more of a piece of shit. Smells like implementing what the cool kids were doing to me.

Agree, unless you were Facebook you at best heard it was cool. Everyone I knew who tried it out immediately went "blecchhh" and switched off or left after they got their company to adopt the technology, heh.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#33

Post by stubish »

hsjqllzlfkf wrote: post #26
leokennis wrote: post #25

Earlier quoted context omitted.

We (as in, my company, not me myself) run large Cassandra clusters in the critical path of bank transaction processing (in the order of 2-25 million payments per day, each requiring a lot of database queries) and it's going pretty well... https://www.youtube.com/watch?v=0QsLU9na2uE But yes, you win some (mainly resilience, availability and disaster avoidance, possibly tunable consistency will help you) you lose some.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

There is always a better solution than Cassandra, until your data will no longer fit on a single server or you actually need guaranteed availability.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#34

Post by mrweasel »

hsjqllzlfkf wrote: post #26
leokennis wrote: post #25

Earlier quoted context omitted.

We (as in, my company, not me myself) run large Cassandra clusters in the critical path of bank transaction processing (in the order of 2-25 million payments per day, each requiring a lot of database queries) and it's going pretty well... https://www.youtube.com/watch?v=0QsLU9na2uE But yes, you win some (mainly resilience, availability and disaster avoidance, possibly tunable consistency will help you) you lose some.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

25 million per day might be a little high for SQLite, especially if they don't spread out evenly over the day. You also get no redundancy or replication, which you might want if your database isn't to grind to a halt during backups.

Arguably Cassandra does sound like a weird choice, but we don't know the specifics of their setup. There's a lot of solutions presented on HN where SQLite and a Java application would have been a better choice and you can say for sure without knowing all the details, I feel like this is past that point.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#35

Post by leokennis »

hsjqllzlfkf wrote: post #26
leokennis wrote: post #25

Earlier quoted context omitted.

We (as in, my company, not me myself) run large Cassandra clusters in the critical path of bank transaction processing (in the order of 2-25 million payments per day, each requiring a lot of database queries) and it's going pretty well... https://www.youtube.com/watch?v=0QsLU9na2uE But yes, you win some (mainly resilience, availability and disaster avoidance, possibly tunable consistency will help you) you lose some.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

Transaction in this case is not a database transaction, but a financial transaction (payment). Per payment, probably somewhere in the order of 50-100 database transactions (although Cassandra does not really have transactions of course, interpret this as read/write actions) will be performed in the course of its processing. So that is 1,875,000,000,000 database actions on busy days. Not a DBA, but for our purposes the scalability and availability of Cassandra works very well.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#36

Post by hardwaresofton »

CHY872 wrote: post #21
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

I have >5 years of experience using Cassandra in production, involving thousands of clusters storing petabytes of data. My conclusion from that time is that Cassandra is simply not robust enough to be a general purpose database (the team are working on it but they're coming from a really rough starting place) - there are lots of ways to cause data corruption, and Cassandra does enough dynamic repairing that it can be…

Thanks for this insight -- this is one of the first time I've heard of someone constrasting FoundationDB and Cassandra which is nice.

> Example of fairly standard Cassandra bug (don't know if present on latest release, certainly was a year or two ago): When you add a new node to the cluster, it 'bootstraps', where it copies ~1/n the data from other nodes. When you are done bootstrapping, it's copied a bunch of data from other nodes, but the other nodes still contain that data. You then run 'cleanups' on the other nodes to remove the (now stale and unusable) data so as to get your disk space back.

Interesting, seems like there is a bunch of little knowledge like this needed to run a service properly... Managed Cassandra has more added value to provide I guess.

> If you accidentally run a cleanup on the new node as it is being bootstrapped, it will succeed, you will delete all the data that's been copied over so far, and Cassandra will _not_ terminate the bootstrap. Everything will be green, but your new node will suddenly be using 0 disk space. When the bootstrap finishes, possibly days later, your cluster will be immediately corrupted due to violated replication guarantees - but only on data that hasn't been read or written over that period, because if it was written it'll be re-replicated, and if it was read Cassandra will silently repair at this time. Repairs resolve the issue, but if you've made this mistake due to scripting, if you get unlucky it's possible to just delete all replicas of some data between repairs.

This seems... really bad -- I don't think I have the skill to run a Cassandra cluster (and not enough use cases to run it as a hobby to find these edges)...

This sounds like the space for a consultancy to make a tidy killing though.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#37

Post by renegade-otter »

avereveard wrote: post #13
pbreit wrote: post #12

Earlier quoted context omitted.

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

Yelp was founded 2004, 2004 postgres was different than 2024 postgres and storage and server options werent as powerful as today either. There were half a million restaurants back then or thereabout, and they wanted to store ratings and comments. That is not something you'd be able to put on one single database in 2004. Why didn't they simplify afterward I can't imagine, but I can see how a business directory at that…

But there was Oracle. I actually worked on a dual MySQL/Oracle setup in 2004, and MySQL seemed like a toy database.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#38

Post by zac23or »

pbreit wrote: post #12
hardwaresofton wrote: post #7

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…

One wonders if Yelp could just run on a basic Postgres setup. It's not too much data, the data is relatively unimportant and the traffic is modest and mostly from US. How do the setups get so complicated?

In my experience, 100% of the time, the use of NoSql (excluding Redis) has nothing to do with the RDBS not being good enough, rather its "solves" another non-technical issue. Examples:

1. In one case, developers hate DBAs and MongoDB is outside the scope of DBAs.

2. A developer starts using CouchDB to include in the curriculum.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#39

Post by Alifatisk »

stubish wrote: post #33
hsjqllzlfkf wrote: post #26

Earlier quoted context omitted.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

There is always a better solution than Cassandra, until your data will no longer fit on a single server or you actually need guaranteed availability.

Is there no other db that offers guaranteed availability?

I remember reading about Discord switching from Cassandra to ScyllaDB I think.

Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture

#40

Post by giovannibonetti »

leokennis wrote: post #35
hsjqllzlfkf wrote: post #26

Earlier quoted context omitted.

To do 2-25 million transactions per day you might as well use SQLite. Sounds like this was a career development push more than anything.

Transaction in this case is not a database transaction, but a financial transaction (payment). Per payment, probably somewhere in the order of 50-100 database transactions (although Cassandra does not really have transactions of course, interpret this as read/write actions) will be performed in the course of its processing. So that is 1,875,000,000,000 database actions on busy days. Not a DBA, but for our purposes th…

There are three extra zeros here
Post reply on HN