Meta quickly detects silent data corruptions at scale • Hacker News Board

Re: Meta quickly detects silent data corruptions at scale

#101

Post by alipitch » Tue, Apr 05, 2022, 10:25 AM UTC

huhtenberg wrote: post #23

notacoward wrote: post #4

To be clear, this is about corruption in the CPU/GPU/memory complex. There's a whole separate set of techniques (some of which I worked on) to detect and correct data corruption on disk.

I'm in the same boat and my takeaway is that the vast majority of a "silent" on-disk corruption actually happens on the way to the storage, i.e. the data gets corrupted in some RAM it passes through and then just ends up being written out in corrupted state. This is because, virtually all modern drives implement per-sector FEC coding, so if a bit does flip on the disk, you will either get back original data (now FEC-…

Slightly off-topic digression: This article discussed "enterprise" grade "silent data corruptions".

What are some recommendations for "personal data storage" grade "silent data corruptions"?

"personal data storage" for my case is I am looking at a wikipedia list below, and then searching through hn comments.

https://en.wikipedia.org/wiki/Comparison_of_file_systems#Blo...> column: Data checksum/ ECC

I found many comments on ZFS, and not so much comments on dm-integrity, BlueStore/Cephfs, and others. So I am thinking of looking into ZFS, but if there are any recommendations, I would like to seek advice.

I am experimenting with Git LFS, git-annex, I like the filesystem UI better, so I am looking for filesystem like solutions.

Re: Meta quickly detects silent data corruptions at scale

#102

Post by southerntofu » Tue, Apr 05, 2022, 10:56 AM UTC

ksec wrote: post #100

fleddr wrote: post #88

Earlier quoted context omitted.

Exactly. And that doesn't even take into account higher res photos and 4K video. I remember the staggering statistic where just Instagram sees 100 million photos added per day, every day. And that was years ago.

That is just Instagram, with photos. Imagine Youtube. And again I have been saying this since ~2015/16, we dont have any meaningful roadmap for cost reduction on storage, whether that is Hot as in NAND, Bulk as in HDD, or Cold as in Optical Disc. I dont see 2TB SSD dropping below $100 in next 5 years, or 10TB HDD below $120. Remember when Google promise infinite Gmail storage?

> we dont have any meaningful roadmap for cost reduction on storage

Then maybe we could:

- stop encouraging users to post shit just so we can track them

- stop tracking them which requires many data points and a lot of processing power (for 0 benefit for the user or society at large)

- stop the copyright non-sense and actually use hyperlinks instead of reuploading the same content 500 times across platforms? maybe even do content-addressed storage (Bittorrent/IPFS) who knows?

Re: Meta quickly detects silent data corruptions at scale

#103

Post by cbetti » Tue, Apr 05, 2022, 12:04 PM UTC

fleddr wrote: post #78

kache_ wrote: post #7

The scale at which Meta operates at really boggles my mind. I work with an ex facebook guy who was on the infra side of things and the numbers he told me.. I couldn't even imagine. And I'm working on the order of magnitude of 100m/h, but still, completely different set of challenges.

I once read that Facebook was opening 2 or 3 massive new data-centers in the US for the purpose of hosting stale content. You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there. It makes me wonder how it can go on like this. Users only keep adding co…

This isn't how scaling works though. Across all applications the hot data growth outpaces the cold.

So if you're designing capacity for exponential growth, the future point at which you stop experiencing exponential growth and only have to worry about roughly linear growth is a much easier problem to solve.

Re: Meta quickly detects silent data corruptions at scale

#104

Post by stewbrew » Tue, Apr 05, 2022, 7:29 PM UTC

nwsm wrote: post #71

stewbrew wrote: post #46

Earlier quoted context omitted.

Meta still redirects meta.com to https://about.facebook.com/meta I don't think it's too soon.

It's too soon to tell if it's a mistake*

It isn't too soon to see that they don't believe in this rebranding themselves. Ergo it was a mistake.