Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta quickly detects silent data corruptions at scale

engineering.fb.com

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Meta quickly detects silent data corruptions at scale

#31

Post by tmn »

tonguez wrote: post #25
tmn wrote: post #19

Earlier quoted context omitted.

Vr is a subset of the ‘metaverse’. The metaverse isn’t really something new. It’s just a rebranding of the portion of our lives that are contained within the digital realm. On top of that there are obviously ideas for how to adapt and grow that space, which is all to be seen

“It’s just a rebranding of the portion of our lives that are contained within the digital realm.” none of my life is “contained” within shitbook

Lol that’s fine. Metaverse isn’t specific to shitbook. You’re on the metaverse via hn. At the end of the day, I think it’s dumb. I’m just iron manning the justification of the rebrand

Re: Meta quickly detects silent data corruptions at scale

#32

Post by [deleted] »

ebbp wrote: post #22
raphaelj wrote: post #17

It would be better if Meta would focus on detecting spam at scale. I put a desk chair on Marketplace last Friday, and got 8 messages that were actually scams. These were trying to "schedule" a Fedex/DHL pickup, and would redirect me to fake branded websites that were requesting my personal details and bank account. This was so obviously fake it baffled me Meta can't detect these automatically. I am also getting multi…

It’s a different team, with a different skillset, that would be responsible for that. Big companies can focus on more than one thing at a time.

[deleted]

Re: Meta quickly detects silent data corruptions at scale

#33

Post by notacoward »

huhtenberg wrote: post #23
notacoward wrote: post #4

To be clear, this is about corruption in the CPU/GPU/memory complex. There's a whole separate set of techniques (some of which I worked on) to detect and correct data corruption on disk.

I'm in the same boat and my takeaway is that the vast majority of a "silent" on-disk corruption actually happens on the way to the storage, i.e. the data gets corrupted in some RAM it passes through and then just ends up being written out in corrupted state. This is because, virtually all modern drives implement per-sector FEC coding, so if a bit does flip on the disk, you will either get back original data (now FEC-…

I can state categorically that bitrot on disk does exist, because that's one of the parts I worked on. It's pretty rare - unfortunately I don't think I can give you the numbers - but across enough exabytes it does happen enough to justify slow scans to detect it.

Re: Meta quickly detects silent data corruptions at scale

#34

Post by lclarkmichalek »

silisili wrote: post #30
kache_ wrote: post #7

The scale at which Meta operates at really boggles my mind. I work with an ex facebook guy who was on the infra side of things and the numbers he told me.. I couldn't even imagine. And I'm working on the order of magnitude of 100m/h, but still, completely different set of challenges.

Same. I remember asking one guy at FB the process to ask for a new server. He said he can't even open a request for anything less than a thousand boxes. The largest fleet I'd worked on at that point was 12... different worlds.

I mean, that's not true in the general case. That'd be incredibly wasteful.

(Work at Meta, mostly on capacity)

Re: Meta quickly detects silent data corruptions at scale

#35

Post by spookthesunset »

raphaelj wrote: post #17

It would be better if Meta would focus on detecting spam at scale. I put a desk chair on Marketplace last Friday, and got 8 messages that were actually scams. These were trying to "schedule" a Fedex/DHL pickup, and would redirect me to fake branded websites that were requesting my personal details and bank account. This was so obviously fake it baffled me Meta can't detect these automatically. I am also getting multi…

At the scale of FB, handling fraud is a non-trivial effort. At any given time there are probably thousands of somewhat well funded fraud teams looking to bypass whatever shiny new countermeasure FB adds to their site.

There is a lot of money to be made from defrauding FB users. This monetary incentive results in criminals investing tons of effort into bypassing anti-fraud stuff. It is a non-stop effort of incremental moves on both parties that will carry on for as long as FB users remain a juicy target.

Re: Meta quickly detects silent data corruptions at scale

#36

Post by Sindisil »

nimbius wrote: post #16
HL33tibCe7 wrote: post #6

Completely off-topic digression: I still think the name change to “Meta” is a big mistake. Subjectively, for some reason I just really dislike the name. More objectively, the branding is very muddled, e.g: serving an “Engineering at Meta” blog post on fb.com. Often with these things it’s just about time; it feels wrong because you’re just not used to the change yet. Maybe that will happen, but it’s been months now. U…

a muddled brand is better than the currently maligned harbinger of misery disinformation and insurrection that Facebook has been mired in. Recruiters at Meta probably appreciate the distance.

How many candidates wouldn't know that Meta == Facebook, at least within the tech spheres?

Re: Meta quickly detects silent data corruptions at scale

#37

Post by mescaline »

lclarkmichalek wrote: post #34
silisili wrote: post #30

Earlier quoted context omitted.

Same. I remember asking one guy at FB the process to ask for a new server. He said he can't even open a request for anything less than a thousand boxes. The largest fleet I'd worked on at that point was 12... different worlds.

I mean, that's not true in the general case. That'd be incredibly wasteful. (Work at Meta, mostly on capacity)

No post body was provided.

Re: Meta quickly detects silent data corruptions at scale

#38

Post by huhtenberg »

notacoward wrote: post #33
huhtenberg wrote: post #23

Earlier quoted context omitted.

I'm in the same boat and my takeaway is that the vast majority of a "silent" on-disk corruption actually happens on the way to the storage, i.e. the data gets corrupted in some RAM it passes through and then just ends up being written out in corrupted state. This is because, virtually all modern drives implement per-sector FEC coding, so if a bit does flip on the disk, you will either get back original data (now FEC-…

I can state categorically that bitrot on disk does exist, because that's one of the parts I worked on. It's pretty rare - unfortunately I don't think I can give you the numbers - but across enough exabytes it does happen enough to justify slow scans to detect it.

How did you know it was a change at rest?

The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the disk to another machine and re-read again. If it doesn't match, only then it's an actual at-rest bitrot... OR it's a drive's firmware bug, because corrupted data must be corrected or it must not be returned at all.

Re: Meta quickly detects silent data corruptions at scale

#39

Post by ryeguy »

mescaline wrote: post #37
lclarkmichalek wrote: post #34

Earlier quoted context omitted.

I mean, that's not true in the general case. That'd be incredibly wasteful. (Work at Meta, mostly on capacity)

Quoted post unavailable.

This is a technical post, stay on topic and stop posting flamewar bait.
Post reply on HN