Earlier quoted context omitted.
Quoted post unavailable.
This is a technical post, stay on topic and stop posting flamewar bait.
It's not flamewar bait unless you turn it into that. It can be a fruitful discussion.
Meta quickly detects silent data corruptions at scale
61–70 of 104 posts
Re: Meta quickly detects silent data corruptions at scale
#62Earlier quoted context omitted.
Same. I remember asking one guy at FB the process to ask for a new server. He said he can't even open a request for anything less than a thousand boxes. The largest fleet I'd worked on at that point was 12... different worlds.
I mean, that's not true in the general case. That'd be incredibly wasteful. (Work at Meta, mostly on capacity)
Curious - do you know what he could have been referring to? SRE position, 6-8 years ago.
Re: Meta quickly detects silent data corruptions at scale
#63Earlier quoted context omitted.
> the name change to "Meta" is a big mistake I think it's too soon to tell. Facebook has really negative brand recognition (from my POV), and who knows, maybe "metaverse" style online interaction is the future. (For the record I'm anti-web3 and indifferent on metaverse communities)
The name Meta dilutes the brand significantly. I bet if you ask people what Meta is, most people outside tech can't tell. But if you ask what Facebook is, 100% of them can. They took a really good brand name and trashed it to the point they needed to rebrand.
I thought that was the point, though. The brand Facebook is well known, but had developed negative associations. So they wanted to start fresh.
Re: Meta quickly detects silent data corruptions at scale
#64Earlier quoted context omitted.
> How did you know it was a change at rest? Because we had the checks for it in flight. Also, more often than not these same blocks had been checked before, and found to be fine. > The only correct way to test for bitrot is to read the data back immediately No, the only correct way is to read it back after some time has passed . Mis-written data is not the same as bitrot. > must be corrected or it must not be returne…
>> The only correct way to test for bitrot is to read the data back immediately > No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot. Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in. > Every e…
> But it can detect that the case when it can't recover.
That is simply not true. For any parity/ECC/FEC/erasure-code scheme carrying M data bits in N (greater than M but less than 2M) total, there must be multiple data patterns that will match the same error checks. That's just mathematics. Also, bear in mind that ECC bits can be corrupted too. This opens up the distinct possibility of something that looks like a correctable error, but the "correction" leads to a wrong result. I've seen such issues in many kinds of storage systems, from low level to high. Anyone who has actually worked in this area, instead of deriving their "expertise" from a quick scan of Wikipedia, would be utterly unsurprised by the idea that disk firmware might do such a thing, or have bugs in their ECC implementation, or not follow a spec.
Whatever the causes, whatever the merely-theoretical probabilities, the fact remains that I've seen these. I've been paged for them. I've done the analyses of possible causes. A bit pattern was written and repeatedly verified over a quite long period of time (ruling out data path issues), then at some point a different bit pattern was read and would persistently be read thereafter. How is that not real bitrot? How does it matter, beyond ruling out everything above the disk level, what the precise causes are? If you can't answer those questions, you're just posting noise.
Re: Meta quickly detects silent data corruptions at scale
#65Earlier quoted context omitted.
>> The only correct way to test for bitrot is to read the data back immediately > No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot. Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in. > Every e…
Didn't you answer your own question above? It's firmware bugs. The disk reported a successful write at block X but it actually wrote block Y. Later you read block Y and you get data X. The block-level ECC codes are consistent. You also stand a low but not zero probability that you requested a read at block X and were served up some other block, again with matching checksums. And of course there's always the possibili…
Fun question: what if a relocation table gets corrupted? And what protection is there against that possibility? You can bet it's not the same ECC as on data blocks. The rest is left as an exercise for the reader. ;)
Re: Meta quickly detects silent data corruptions at scale
#66Re: Meta quickly detects silent data corruptions at scale
#67Earlier quoted context omitted.
I mean, that's not true in the general case. That'd be incredibly wasteful. (Work at Meta, mostly on capacity)
Curious - do you know what he could have been referring to? SRE position, 6-8 years ago.
A bit before my time, but maybe a physical capacity order, to a non fungible pool. The infra is quite different now (mostly to avoid this kind of inefficiency)
Re: Meta quickly detects silent data corruptions at scale
#68In a fleet of 100,000 machines, there will always be some clear failures... When the machine has 2x the number of segfaults of any other machine in the fleet, you send it for repairs and someone replaces the motherboard, ram and CPU... easy! But the painful ones are the 'subtle' failures. Why does machine PABL12 sometimes give NaN as a result while all 99,999 machines return sensible numbers? But all the burn in hard…
> When the machine has 2x the number of segfaults of any other machine in the fleet, you send it for repairs
At that scale, it's quite likely sent to repair automatically and whoever's on call just gets a notification.
Re: Meta quickly detects silent data corruptions at scale
#69Earlier quoted context omitted.
The name Meta dilutes the brand significantly. I bet if you ask people what Meta is, most people outside tech can't tell. But if you ask what Facebook is, 100% of them can. They took a really good brand name and trashed it to the point they needed to rebrand.
I thought that was the point, though. The brand Facebook is well known, but had developed negative associations. So they wanted to start fresh.
Two sides of the same coin.
Re: Meta quickly detects silent data corruptions at scale
#70Earlier quoted context omitted.
I will always say VR, I will never say "metaverse". Their branding move was bold, yet unconvincing.
Vr is a subset of the ‘metaverse’. The metaverse isn’t really something new. It’s just a rebranding of the portion of our lives that are contained within the digital realm. On top of that there are obviously ideas for how to adapt and grow that space, which is all to be seen
Based