This FAST '08 paper "Parity Lost and Parity Regained" is still the one I pull out and show people if they seem to be under-imagining all the crimes an HDD can do.
https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
Disks Lie: Building a WAL that actually survives
11–20 of 53 posts
Re: Disks Lie: Building a WAL that actually survives
#12People consistently underestimate the many ways in which storage can and will fail in the wild.
The most vexing storage failure is phantom writes. A disk read returns a "valid" page, just not the last written/fsync-ed version of that page. Reliably detecting this case is very expensive, particularly on large storage volumes, so it is rarely done for storage where performance is paramount.
Re: Disks Lie: Building a WAL that actually survives
#13I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
Re: Disks Lie: Building a WAL that actually survives
#14I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
[deleted]
Re: Disks Lie: Building a WAL that actually survives
#15> Conclusion > A production-grade WAL isn't just code, it's a contract. I hate that I'm now suspicious of this formulation.
You’re not insane. This is definitely AI.
Re: Disks Lie: Building a WAL that actually survives
#16I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
it's not just a good idea for raid
Re: Disks Lie: Building a WAL that actually survives
#17People consistently underestimate the many ways in which storage can and will fail in the wild. The most vexing storage failure is phantom writes. A disk read returns a "valid" page, just not the last written/fsync-ed version of that page. Reliably detecting this case is very expensive, particularly on large storage volumes, so it is rarely done for storage where performance is paramount.
Not that uncommon failure mode for some SSDs, unclean shutdown is like a dice roll for some of them: maybe you get what you wrote five seconds ago, maybe you get a snapshot of a couple hours ago.
Re: Disks Lie: Building a WAL that actually survives
#18Check out Parity Lost and Parity Regained and Characteristics, Impact, and Tolerance of Partial Disk Failures (which this blog indirectly cites) if you'd like authoritative reading on the topic.
https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...
Re: Disks Lie: Building a WAL that actually survives
#19This FAST '08 paper "Parity Lost and Parity Regained" is still the one I pull out and show people if they seem to be under-imagining all the crimes an HDD can do. https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
Seconded.
Re: Disks Lie: Building a WAL that actually survives
#20I worked with a greybeard that instilled in me that when we were about to do some RAID maintenance that we would always run sync twice. The second to make sure it immediately returns. And I added a third for my own anxiety.
it's not just a good idea for raid
Oh definitely not, I do it on every system that I've needed it to be synced before I did something. We were just working at a place that had 2k+ physical servers with 88 drives each in RAID6, so that was our main concern back then.
I have been passing my anxieties about hardrives to junior engineers for a decade now.