Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta quickly detects silent data corruptions at scale

engineering.fb.com

Post reply on HN
Print view View original

81–90 of 104 posts

Re: Meta quickly detects silent data corruptions at scale

#81

Post by jonah-archive »

notacoward wrote: post #65
jeffbee wrote: post #58

Earlier quoted context omitted.

Didn't you answer your own question above? It's firmware bugs. The disk reported a successful write at block X but it actually wrote block Y. Later you read block Y and you get data X. The block-level ECC codes are consistent. You also stand a low but not zero probability that you requested a read at block X and were served up some other block, again with matching checksums. And of course there's always the possibili…

Fun question: what if a relocation table gets corrupted? And what protection is there against that possibility? You can bet it's not the same ECC as on data blocks. The rest is left as an exercise for the reader. ;)

uh oh, I recognize this one. Love to have a file corrupt after months at rest with no access logged and no mtime changes because a file on a neighboring track needed rewriting (SMR, of course).

Re: Meta quickly detects silent data corruptions at scale

#82

Post by wiredfool »

doix wrote: post #20
Hnrobert42 wrote: post #12

Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.

Depending on why you're using a VPN, you can just pay for a tiny VPS from ovh/hetzner, setup wireguard and use that as your VPN. Obviously don't do anything illegal, since everything is going through a server that is directly tied your credit card. But for privacy/security, it's good enough (for me anyway). I'm guessing it's luck of the draw if your IP has been blacklisted or not, but I've not had any issues the last…

Hetzner commonly triggers cloudflare's bot detection, and there are some things that just refuse to talk to it’s ip space.

Re: Meta quickly detects silent data corruptions at scale

#83

Post by huhtenberg »

notacoward wrote: post #64
huhtenberg wrote: post #50

Earlier quoted context omitted.

>> The only correct way to test for bitrot is to read the data back immediately > No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot. Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in. > Every e…

> But it can detect that the case when it can't recover. That is simply not true. For any parity/ECC/FEC/erasure-code scheme carrying M data bits in N (greater than M but less than 2M) total, there must be multiple data patterns that will match the same error checks. That's just mathematics. Also, bear in mind that ECC bits can be corrupted too. This opens up the distinct possibility of something that looks like a co…

> That is simply not true.

It indeed is not. Had to reread the theory and I stand corrected, RS-style ECC can't detect errors in excess of the redundancy count.

> How is that not real bitrot?

It is and I can see how it can happen.

> How does it matter, beyond ruling out everything above the disk level, what the precise causes are?

It would've mattered if a drive could detect on-disk bitrot reliably, which was what the stats I worked with (also in exabytes, funnily enough) and the IEEE papers I read led me to believe.

For what it's worth, you won. Hats off.

Re: Meta quickly detects silent data corruptions at scale

#84

Post by tomrod »

tmn wrote: post #31
tonguez wrote: post #25

Earlier quoted context omitted.

“It’s just a rebranding of the portion of our lives that are contained within the digital realm.” none of my life is “contained” within shitbook

Lol that’s fine. Metaverse isn’t specific to shitbook. You’re on the metaverse via hn. At the end of the day, I think it’s dumb. I’m just iron manning the justification of the rebrand

No, you and I and parent commenter are on the world wide web on HN.

I see no 3d Second Life models interacting.

Re: Meta quickly detects silent data corruptions at scale

#85

Post by philjohn »

criddell wrote: post #77
Hnrobert42 wrote: post #12

Interestingly, this site fails ungracefully (HTTP error code 500) when I try to visit from NordVPN, even after cycling through a few IP addresses. I’m noticing more and more sites block all VPN track. I get why, but it’s not good.

Do you know how they detect VPN traffic? When I'm traveling I'll often pipe my traffic through a VPN on my home network. I have had some weird failures but I've usually assumed that it was due to an unreliable hotspot I'm using. Now I'm wondering if using a VPN is the real problem...

It'll be due to the exit nodes for the VPN having been put on a blocklist.

If you're tunneling through your home network it's unlikely to cause problems, unless you've been doing nefarious things from your home IP and that has also ended up on a blocklist.

Re: Meta quickly detects silent data corruptions at scale

#86

Post by darkwater »

jreese wrote: post #79
fleddr wrote: post #78

Earlier quoted context omitted.

I once read that Facebook was opening 2 or 3 massive new data-centers in the US for the purpose of hosting stale content. You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there. It makes me wonder how it can go on like this. Users only keep adding co…

There's no meaningful benefit to dedicating any amount of DC equipment just to stale content. Those are spindles (and networks) that could be taking meaningful hot reads and writes, and colocating stale and hot data is generally a better use of capacity than concentrating hot data in fewer locations.

What you say totally makes sense but still, even if new media takes more storage space, the accumulated stale date in the long run will win.

Or maybe I'm underestimating how much space newer material needs?

Re: Meta quickly detects silent data corruptions at scale

#87

Post by notacoward »

huhtenberg wrote: post #83
notacoward wrote: post #64

Earlier quoted context omitted.

> But it can detect that the case when it can't recover. That is simply not true. For any parity/ECC/FEC/erasure-code scheme carrying M data bits in N (greater than M but less than 2M) total, there must be multiple data patterns that will match the same error checks. That's just mathematics. Also, bear in mind that ECC bits can be corrupted too. This opens up the distinct possibility of something that looks like a co…

> That is simply not true. It indeed is not. Had to reread the theory and I stand corrected, RS-style ECC can't detect errors in excess of the redundancy count. > How is that not real bitrot? It is and I can see how it can happen. > How does it matter, beyond ruling out everything above the disk level, what the precise causes are? It would've mattered if a drive could detect on-disk bitrot reliably, which was what th…

Thank you for an interesting (despite being contentious) conversation.

Re: Meta quickly detects silent data corruptions at scale

#88

Post by fleddr »

darkwater wrote: post #86
jreese wrote: post #79

Earlier quoted context omitted.

There's no meaningful benefit to dedicating any amount of DC equipment just to stale content. Those are spindles (and networks) that could be taking meaningful hot reads and writes, and colocating stale and hot data is generally a better use of capacity than concentrating hot data in fewer locations.

What you say totally makes sense but still, even if new media takes more storage space, the accumulated stale date in the long run will win. Or maybe I'm underestimating how much space newer material needs?

Exactly. And that doesn't even take into account higher res photos and 4K video. I remember the staggering statistic where just Instagram sees 100 million photos added per day, every day. And that was years ago.

Re: Meta quickly detects silent data corruptions at scale

#89

Post by jreese »

darkwater wrote: post #86
jreese wrote: post #79

Earlier quoted context omitted.

There's no meaningful benefit to dedicating any amount of DC equipment just to stale content. Those are spindles (and networks) that could be taking meaningful hot reads and writes, and colocating stale and hot data is generally a better use of capacity than concentrating hot data in fewer locations.

What you say totally makes sense but still, even if new media takes more storage space, the accumulated stale date in the long run will win. Or maybe I'm underestimating how much space newer material needs?

The amount of ephemeral content (stories, expiring content) being created/shared has a larger impact on capacity and provisioning than you might expect.

Re: Meta quickly detects silent data corruptions at scale

#90

Post by gitfan86 »

fleddr wrote: post #88
darkwater wrote: post #86

Earlier quoted context omitted.

What you say totally makes sense but still, even if new media takes more storage space, the accumulated stale date in the long run will win. Or maybe I'm underestimating how much space newer material needs?

Exactly. And that doesn't even take into account higher res photos and 4K video. I remember the staggering statistic where just Instagram sees 100 million photos added per day, every day. And that was years ago.

Instagram does not guarantee photo quality. They can resize photos anytime they want. Eventually they turn 10 year old photos that have not been viewed in 9 years into thumbnails or just delete them.
Post reply on HN