Meta quickly detects silent data corruptions at scale
engineering.fb.com
Meta quickly detects silent data corruptions at scale
1–10 of 104 posts
Re: Meta quickly detects silent data corruptions at scale
#2In a fleet of 100,000 machines, there will always be some clear failures... When the machine has 2x the number of segfaults of any other machine in the fleet, you send it for repairs and someone replaces the motherboard, ram and CPU... easy!
But the painful ones are the 'subtle' failures. Why does machine PABL12 sometimes give NaN as a result while all 99,999 machines return sensible numbers? But all the burn in hardware tests pass...
The solution was to simply exclude any machines that were outliers. Anything in the top or bottom 0.01% for any metric simply exclude that machine from future workloads.
Sure, in most cases there was nothing wrong with the hardware, but when you're spending hours debugging some fault caused by a sometimes-bad floating point unit on one core of one machine out of 100,000, you're just wasting your time. By auto-banning outliers, the machine will end up doing some other task where data consistency matters less.
Re: Meta quickly detects silent data corruptions at scale
#3Some might enjoy this old Cloudflare debugging story about random crashes in production.
https://blog.cloudflare.com/however-improbable-the-story-of-...
Re: Meta quickly detects silent data corruptions at scale
#4To be clear, this is about corruption in the CPU/GPU/memory complex. There's a whole separate set of techniques (some of which I worked on) to detect and correct data corruption on disk.
Re: Meta quickly detects silent data corruptions at scale
#5Some might enjoy this old Cloudflare debugging story about random crashes in production. https://blog.cloudflare.com/however-improbable-the-story-of-...
Thanks for sharing.
Re: Meta quickly detects silent data corruptions at scale
#6Completely off-topic digression: I still think the name change to “Meta” is a big mistake. Subjectively, for some reason I just really dislike the name. More objectively, the branding is very muddled, e.g: serving an “Engineering at Meta” blog post on fb.com.
Often with these things it’s just about time; it feels wrong because you’re just not used to the change yet. Maybe that will happen, but it’s been months now. Usually with these changes I change my mind quicker than that.
Re: Meta quickly detects silent data corruptions at scale
#7The scale at which Meta operates at really boggles my mind. I work with an ex facebook guy who was on the infra side of things and the numbers he told me.. I couldn't even imagine. And I'm working on the order of magnitude of 100m/h, but still, completely different set of challenges.
Re: Meta quickly detects silent data corruptions at scale
#8Re: Meta quickly detects silent data corruptions at scale
#9Content seems interesting but the generic corporate image at the top, crap font and off-black low contrast text colour is getting on my nerves.
Re: Meta quickly detects silent data corruptions at scale
#10Content seems interesting but the generic corporate image at the top, crap font and off-black low contrast text colour is getting on my nerves.
Reader mode works great on mobile Safari.