Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
1–10 of 53 posts
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#2Can second that this is a very solid and reliable way to do any kind of migration, not just recovering from corruption. When possible being able to insert your own code in the middle of the transfer and shunt bad data to a dead letter queue is a life saver sometimes because the last thing you want to have happen is the xfer get stuck when it hits a pothole.
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#3I am excited to read about system recovery in most cases.
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#4Source: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... (Jan 30, 2023)
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#5Despite best efforts, the shard and all of its data was lost. However due to their forward-thinking strategy it was limited to only 3-star and lower reviews for Yelp Premium clients!
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#6What is a bit unclear which isn't addressed in the original blog post is how they actually were able to successfully scan over all the data and copy it to the new cluster, in spite of corrupt sstables. CDC would not handle back filling historical data. Would expect that to fail for certain token ranges. Perhaps they ended up deciding to discard them?
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#7For those that care about how it got corrupted:
https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan...
> The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key
This was just... a bug in Cassandra? Is there anyone that can shed light on this? There seem to be plenty of people using Cassandra at scale -- is constantly repairing it normal practice?
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#8Love reading this, it's a story about how a back of house team got to have an impact on the business in a more immediate way than they usually get to. Data teams have a longer time horizon for when they succeed or fail than the glamorous lives of backend - let alone frontend! - engineering teams, which makes it harder to recognize the wins in the moment.
To any Yelp data engineers who might happen to read - good work, and it's a good testament to the platform you provide.
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#9For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…
Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very not realisable, would consume huge volumes of memory (especially during repairs), bandwidth (doing a repair is very chatty as it has to sync lots of data) and disk space (tombstoning meant deleted records take up space until compaction runs), and was generally not much fun to manage, and it was difficult to hire people who knew much about it to do so. I would not build a solution myself using it going forward. We also had to periodically (weekly) do "full" repairs to work around Cassandra bugs, silent data corruption etc...
Re: Yelp rebuilds corrupted Cassandra cluster using its data streaming architecture
#10For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassan... > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can she…
Running regular incremental repairs is the norm, as nodes will from time to time have trouble talking to each other due to real world network reasons, or will go down, for things like OS patching. We had a (daily) cron job for it. I come from the software side not the DBA side of things but my main advice from running Cassandra at scale in production (it was part of an Apigee stack) is don't basically! It was very no…
Can recommend https://cassandra-reaper.io/ for most of the management stuff you're mentioning. Still not free though, running Cassandra requires (some) effort in my experience.