Meta quickly detects silent data corruptions at scale • Hacker News Board

Re: Meta quickly detects silent data corruptions at scale

#51

Post by danuker » Mon, Apr 04, 2022, 4:42 PM UTC

londons_explore wrote: post #49

PragmaticPulp wrote: post #48

Earlier quoted context omitted.

I've been on the other side of this (threat analysis, not Facebook). Known VPN-associated IP addresses were far more likely to be associated with abuse than average. Not just a little bit, but approaching 2 orders of magnitude worse in our case. It's not even close. It's too bad for the people who need to use public VPN services for whatever reason, but until we have perfect bot/abuse detection, banning VPN, Tor, and…

But it's hard to 'abuse' reading a blog post...

There's denial of service, which wastes server resources, reducing the accessibility to humans interested in the content.

Re: Meta quickly detects silent data corruptions at scale

#52

Post by huhtenberg » Mon, Apr 04, 2022, 4:46 PM UTC

booi wrote: post #44

huhtenberg wrote: post #38

Earlier quoted context omitted.

How did you know it was a change at rest ? The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the…

ZFS and other checksumming file systems can detect bit rot in data at rest. When data is read back, that sector is checksummed and compared against when it was written before returning the request. You can periodically scrub the entire pool to find and even fix these issues (in a pool with redundancy)

Sure. However the main purpose of scrubbing is to flush out deteriorating media and to prompt the drive to relocate salvageable sectors and to report completely dead ones.

Re: Meta quickly detects silent data corruptions at scale

#53

Post by jeffbee » Mon, Apr 04, 2022, 4:49 PM UTC

londons_explore wrote: post #2

In a fleet of 100,000 machines, there will always be some clear failures... When the machine has 2x the number of segfaults of any other machine in the fleet, you send it for repairs and someone replaces the motherboard, ram and CPU... easy! But the painful ones are the 'subtle' failures. Why does machine PABL12 sometimes give NaN as a result while all 99,999 machines return sensible numbers? But all the burn in hard…

Was pabl12 an actual bad machine? Sounds somehow plausible, as if I heard of it before.

It was an annoying struggle trying to raise the visibility of broken CPUs during my years at Google SRE. The SRE org and the rest of the software side of Tech Infra resisted the whole concept, even though it was well-known among platforms hardware eng. The process for taking a known-bad machine out of service involved 1) the machine being reported independently by three different teams; 2) the machine continuing to be in service for days or weeks, at the leisure of some very asynchronous automation; and 3) the machine being returned immediately to service because it passed all of the cursory checks during reinstall. Really irritating. Consequently every major service had to maintain their own private blacklist.

It's nice to see that some influential people on the software side are starting to come around, with papers like "Cores That Don't Count" etc, but man they could have been on this boat a decade ago.

Re: Meta quickly detects silent data corruptions at scale

#54

Post by PTOB » Mon, Apr 04, 2022, 4:52 PM UTC

I work on the physical side; building hyperscale datacenters. You guys should try your hand at managing errors in that system. You've got it all: memory leaks, thermal overloads, misallocated heaps, pipes with strong type requirements, dropped packets ... you name it.

Re: Meta quickly detects silent data corruptions at scale

#55

Post by nowherebeen » Mon, Apr 04, 2022, 5:18 PM UTC

nwsm wrote: post #11

HL33tibCe7 wrote: post #6

Completely off-topic digression: I still think the name change to “Meta” is a big mistake. Subjectively, for some reason I just really dislike the name. More objectively, the branding is very muddled, e.g: serving an “Engineering at Meta” blog post on fb.com. Often with these things it’s just about time; it feels wrong because you’re just not used to the change yet. Maybe that will happen, but it’s been months now. U…

> the name change to "Meta" is a big mistake I think it's too soon to tell. Facebook has really negative brand recognition (from my POV), and who knows, maybe "metaverse" style online interaction is the future. (For the record I'm anti-web3 and indifferent on metaverse communities)

The name Meta dilutes the brand significantly. I bet if you ask people what Meta is, most people outside tech can't tell. But if you ask what Facebook is, 100% of them can. They took a really good brand name and trashed it to the point they needed to rebrand.

Re: Meta quickly detects silent data corruptions at scale

#56

Post by summerlight » Mon, Apr 04, 2022, 5:23 PM UTC

lclarkmichalek wrote: post #34

silisili wrote: post #30

Earlier quoted context omitted.

Same. I remember asking one guy at FB the process to ask for a new server. He said he can't even open a request for anything less than a thousand boxes. The largest fleet I'd worked on at that point was 12... different worlds.

I mean, that's not true in the general case. That'd be incredibly wasteful. (Work at Meta, mostly on capacity)

Yeah, I guess it probably is some sort of criteria that requires a formal request? The resource under that number might be automatically provisioned via "freebie quota" per team. I don't work for Meta but I believe most of the big techs work in a similar way.

Re: Meta quickly detects silent data corruptions at scale

#57

Post by mjevans » Mon, Apr 04, 2022, 5:24 PM UTC

jeffbee wrote: post #53

londons_explore wrote: post #2

In a fleet of 100,000 machines, there will always be some clear failures... When the machine has 2x the number of segfaults of any other machine in the fleet, you send it for repairs and someone replaces the motherboard, ram and CPU... easy! But the painful ones are the 'subtle' failures. Why does machine PABL12 sometimes give NaN as a result while all 99,999 machines return sensible numbers? But all the burn in hard…

Was pabl12 an actual bad machine? Sounds somehow plausible, as if I heard of it before. It was an annoying struggle trying to raise the visibility of broken CPUs during my years at Google SRE. The SRE org and the rest of the software side of Tech Infra resisted the whole concept, even though it was well-known among platforms hardware eng. The process for taking a known-bad machine out of service involved 1) the machi…

Reminds me of the typical story of someone with a complete damage protection plan and a flaky device. Take it in for repairs, passes all the tests, but they know it's funky, so snap it in half or otherwise completely wreck it right in front of the tech and demand that repair.

Re: Meta quickly detects silent data corruptions at scale

#58

Post by jeffbee » Mon, Apr 04, 2022, 5:24 PM UTC

huhtenberg wrote: post #50

notacoward wrote: post #42

Earlier quoted context omitted.

> How did you know it was a change at rest? Because we had the checks for it in flight. Also, more often than not these same blocks had been checked before, and found to be fine. > The only correct way to test for bitrot is to read the data back immediately No, the only correct way is to read it back after some time has passed . Mis-written data is not the same as bitrot. > must be corrected or it must not be returne…

>> The only correct way to test for bitrot is to read the data back immediately > No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot. Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in. > Every e…

Didn't you answer your own question above? It's firmware bugs. The disk reported a successful write at block X but it actually wrote block Y. Later you read block Y and you get data X. The block-level ECC codes are consistent. You also stand a low but not zero probability that you requested a read at block X and were served up some other block, again with matching checksums. And of course there's always the possibility that your firmware simply has a bug in the code checker.

The paper "Parity Lost and Parity Regained" assigns a probability of 1.88e−5 to misdirected writes bugs among disks, so if you have a warehouse full of disks you now have this nightmare.

Re: Meta quickly detects silent data corruptions at scale

#59

Post by Melatonic » Mon, Apr 04, 2022, 5:40 PM UTC

PTOB wrote: post #54

I work on the physical side; building hyperscale datacenters. You guys should try your hand at managing errors in that system. You've got it all: memory leaks, thermal overloads, misallocated heaps, pipes with strong type requirements, dropped packets ... you name it.

I would probably be overwhelmed just managing the infrastructure for your monitoring systems and infrastructure is my main thing :-D

Re: Meta quickly detects silent data corruptions at scale

#60

Post by Melatonic » Mon, Apr 04, 2022, 5:44 PM UTC

pavlov wrote: post #43

Sindisil wrote: post #36

Earlier quoted context omitted.

How many candidates wouldn't know that Meta == Facebook, at least within the tech spheres?

The point of the rebrand is not to hide the association, but to make people think Meta > Facebook. I suspect it does help because a lot of hiring goes on for Quest, Instagram, Portal, WhatsApp, Workplace — you can talk to candidates about those specific products rather than make them think of grandpa posting right-wing memes on the blue app.

Yea this is exactly it. I would never want to work for Facebook but I might want to work for Oculus. Even though I know Meta is just Facebook in new clothing it might (for some) psychologically distance it enough.

That being said with all of the data Facebook has from Facebook itself I would imagine Suckerberg must know that for the long term success of his different businesses they do have to distance the smaller ones a bit more than they are now - personally I am hoping we see a major fall in Facebooks value and influence and things like Oculus are spun-off to be as autonomous as possible.