Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta quickly detects silent data corruptions at scale

engineering.fb.com

Post reply on HN
Print view View original

41–50 of 104 posts

Re: Meta quickly detects silent data corruptions at scale

#41

Post by grumbel »

tonguez wrote: post #25
tmn wrote: post #19

Earlier quoted context omitted.

Vr is a subset of the ‘metaverse’. The metaverse isn’t really something new. It’s just a rebranding of the portion of our lives that are contained within the digital realm. On top of that there are obviously ideas for how to adapt and grow that space, which is all to be seen

“It’s just a rebranding of the portion of our lives that are contained within the digital realm.” none of my life is “contained” within shitbook

The Metaverse, at least in theory, is the connection of all aspects of your digital existence into a seamless whole. It wouldn't be limited to Facebook, it would give you a digital identity you can freely carry between websites, VR environments and devices.

In reality of course none of that exists and Meta has so far not shown how they plan to accomplish that. Worse yet, Facebook is directly responsible for making things not seamless on the Internet and in VR. So I don't have much hope (or fear) of them actually being successful in building that. But their vision is a lot broader than just Facebook in VR.

Re: Meta quickly detects silent data corruptions at scale

#42

Post by notacoward »

huhtenberg wrote: post #38
notacoward wrote: post #33

Earlier quoted context omitted.

I can state categorically that bitrot on disk does exist, because that's one of the parts I worked on. It's pretty rare - unfortunately I don't think I can give you the numbers - but across enough exabytes it does happen enough to justify slow scans to detect it.

How did you know it was a change at rest ? The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the…

> How did you know it was a change at rest?

Because we had the checks for it in flight. Also, more often than not these same blocks had been checked before, and found to be fine.

> The only correct way to test for bitrot is to read the data back immediately

No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot.

> must be corrected or it must not be returned

Every error-correction technique has a limit to how many simultaneous errors it can correct. Beyond that, bits can be flipped in a way that seems valid but in fact is not (detectable by cross-checking with other erasure-coded fragments of the same block on other machines). Just because you haven't seen it doesn't mean it doesn't happen. As I said, and as others have said many times, with sufficient scale and time even the most unlikely scenarios become almost inevitable. Why do you persist in telling me I didn't see what I saw with my own eyes? Are you assuming that my thirty years in storage gave me less understanding or insight regarding these issues than whatever experience (if any) you have?

Re: Meta quickly detects silent data corruptions at scale

#43

Post by pavlov »

Sindisil wrote: post #36
nimbius wrote: post #16

Earlier quoted context omitted.

a muddled brand is better than the currently maligned harbinger of misery disinformation and insurrection that Facebook has been mired in. Recruiters at Meta probably appreciate the distance.

How many candidates wouldn't know that Meta == Facebook, at least within the tech spheres?

The point of the rebrand is not to hide the association, but to make people think Meta > Facebook.

I suspect it does help because a lot of hiring goes on for Quest, Instagram, Portal, WhatsApp, Workplace — you can talk to candidates about those specific products rather than make them think of grandpa posting right-wing memes on the blue app.

Re: Meta quickly detects silent data corruptions at scale

#44

Post by booi »

huhtenberg wrote: post #38
notacoward wrote: post #33

Earlier quoted context omitted.

I can state categorically that bitrot on disk does exist, because that's one of the parts I worked on. It's pretty rare - unfortunately I don't think I can give you the numbers - but across enough exabytes it does happen enough to justify slow scans to detect it.

How did you know it was a change at rest ? The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the…

ZFS and other checksumming file systems can detect bit rot in data at rest. When data is read back, that sector is checksummed and compared against when it was written before returning the request.

You can periodically scrub the entire pool to find and even fix these issues (in a pool with redundancy)

Re: Meta quickly detects silent data corruptions at scale

#46

Post by stewbrew »

nwsm wrote: post #11
HL33tibCe7 wrote: post #6

Completely off-topic digression: I still think the name change to “Meta” is a big mistake. Subjectively, for some reason I just really dislike the name. More objectively, the branding is very muddled, e.g: serving an “Engineering at Meta” blog post on fb.com. Often with these things it’s just about time; it feels wrong because you’re just not used to the change yet. Maybe that will happen, but it’s been months now. U…

> the name change to "Meta" is a big mistake I think it's too soon to tell. Facebook has really negative brand recognition (from my POV), and who knows, maybe "metaverse" style online interaction is the future. (For the record I'm anti-web3 and indifferent on metaverse communities)

Meta still redirects meta.com to https://about.facebook.com/meta

I don't think it's too soon.

Re: Meta quickly detects silent data corruptions at scale

#47

Post by someotherperson »

chaorace wrote: post #26
Hnrobert42 wrote: post #12

Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.

I've noticed this is quite popular among the kings of cargo-cult security: banking websites. I can only hope the proliferation of VPN-gating is more contained compared to the recent (banking-led) upswing in Android root-checks. This type of security theater can be easily bypassed by any determined attacker and thus only serves to deter honest users.

> This type of security theater can be easily bypassed by any determined attacker and thus only serves to deter honest users.

To play devil's advocate, the large amount of attackers aren't really determined. They're just fishing for easy targets. If you check the logs on a VPS you'll see an endless stream of people trying to exploit things like Wordpress 24/7 on your brand new VPS that has nothing but a html landing page.

With banks, I imagine they have a compliance check list they have to tick off to make sure that -- if and when a successful attack happens -- their insurance would pay out. If they haven't taken simple steps like blocking VPNs it could lead to the insurance company claiming negligence.

Re: Meta quickly detects silent data corruptions at scale

#48

Post by PragmaticPulp »

Hnrobert42 wrote: post #12

Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.

I've been on the other side of this (threat analysis, not Facebook).

Known VPN-associated IP addresses were far more likely to be associated with abuse than average. Not just a little bit, but approaching 2 orders of magnitude worse in our case. It's not even close.

It's too bad for the people who need to use public VPN services for whatever reason, but until we have perfect bot/abuse detection, banning VPN, Tor, and proxy services is far and away the most effective tool for cutting down on abuse.

Re: Meta quickly detects silent data corruptions at scale

#49

Post by londons_explore »

PragmaticPulp wrote: post #48
Hnrobert42 wrote: post #12

Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.

I've been on the other side of this (threat analysis, not Facebook). Known VPN-associated IP addresses were far more likely to be associated with abuse than average. Not just a little bit, but approaching 2 orders of magnitude worse in our case. It's not even close. It's too bad for the people who need to use public VPN services for whatever reason, but until we have perfect bot/abuse detection, banning VPN, Tor, and…

But it's hard to 'abuse' reading a blog post...

Re: Meta quickly detects silent data corruptions at scale

#50

Post by huhtenberg »

notacoward wrote: post #42
huhtenberg wrote: post #38

Earlier quoted context omitted.

How did you know it was a change at rest ? The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the…

> How did you know it was a change at rest? Because we had the checks for it in flight. Also, more often than not these same blocks had been checked before, and found to be fine. > The only correct way to test for bitrot is to read the data back immediately No, the only correct way is to read it back after some time has passed . Mis-written data is not the same as bitrot. > must be corrected or it must not be returne…

>> The only correct way to test for bitrot is to read the data back immediately

> No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot.

Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in.

> Every error-correction technique has a limit to how many simultaneous errors it can correct.

But it can detect that the case when it can't recover. Which is why it will either produce a correct output or an error.

> As I said, and as others have said many times, with sufficient scale and time even the most unlikely scenarios become almost inevitable.

This is not an argument if it goes against how things actually work.

> Why do you persist in telling me I didn't see what I saw with my own eyes?

I am merely curious in your exact testing technique, because at-rest bitrot is vanishingly impossible, even at the exabyte scale. For it to happen, the data and its ECC (7-11% of the data size) need to be both corrupted in a coordinated way. That is exceedingly unlikely. Especially in the context of academic papers that found that on-disk corruption is nearly always clustered and is either small scale or full-sector failures.

So when you say you ran into a lot of these cases, it's only natural to ask for details. And "scale" is not a detail.

> Are you assuming that my thirty years in storage gave me less understanding or insight regarding these issues than whatever experience (if any) you have?

I have no way to tell. But given your experience, can you explain how at-rest bitrot, should it occur, can seep through the on-disk error correction? I am not talking about raid-style setups, just the banal ECC record in a disk sector [1].

[1] https://en.wikipedia.org/wiki/Advanced_Format#Overview (linking to Advanced Format, because it has a diagram)

Post reply on HN