In OpenZFS and Btrfs, everyone was just guessing

Most of these screeds come down to a fundamental error: trusting a single filesystem with your important data.

If you trust a single filesystem with your data, you're wrong. It doesn't matter how fancy it is, how many parity blocks it has, or how many disks it's backed by: a single logical filesystem definitionally cannot be redundant. If you pretend it is, it will blow up in your face one day, and it will be nobody's fault but yours.

My backup is a drawer full of individual hard drives. Unlike any filesystem that ever has been or ever will be invented, that drawer cannot corrupt my data.

There was, actually, a test, but it turns out to be rather hard to make a deterministic test to reproduce a rare race condition without sticking your fingers in at runtime and forcing the "wrong" ordering.

> So user's are at fault for trusting a filesystem that advertises itself as being reliable?

Yes, absolutely 100% at fault. There are many failure modes here beyond corruption: human error, electrical surges, flooding, fire. Hell, somebody could break into your house and steal the NAS!

Filesystem bugs aren't a big deal. If they are a big deal to you, it's because you're trusting a system with a single point of failure, and the consequences of that inevitable failure are not acceptable to you. Stop doing that.

If I could choose between introducing a random filesystem bug and a random wifi driver bug into the kernel on my laptop, I'd take the filesystem bug every day of the week and twice on Sunday. Reinstalling my laptop is at worst a minor inconvenience that takes me an hour, and I can always use a different filesystem until the bug is fixed.

Welcome to fucking software. I updated Mac OS X one time (or maybe it was just iTunes) and it deleted my home directory (remember that one)? I didn't open the mail promptly and missed the recall notice that my car's antilock brakes had a failure mode that was catastrophic in snow, and then I drove in the snow and then couldn't stop the car. I flew on this new model of Boeing jet a couple years ago, and it had a bug and crashed and I died.

OK, one of those is not true but this is how ALL software of any complexity is. I'm also old and ornery like the OP and I don't like it and if I get drunk I will talk hella shit about it but it's just a fact.

Software sucks. There was indeed a time when we (collectively) did a lot more to try to prove it worked before shipping it. But then:

a.) the money people realized it made money if it mostly worked, and made less money if we went all formal-proofy on it

b.) the software people realized that all the specs and architecture documents and fuzz testing and automated semantic analysis and literal actual voodoo dolls all did totally help, BUT...

(you know, they help ... BUT probably wouldn't have helped in this case, or lots of other cases...)

So this is just the way software is in almost every walk of life. ZFS is amazing, one of the most life-improving software technologies for me; up there with antibiotics, and UTF-8.

Yes, they had a pretty amazing bug, too. So yeah, "... at least for now, there's simply no way to reliably detect bit rot and other data integrity issues and be assured they can be remedied."

Which is just like it ALWAYS HAS BEEN and ALWAYS WILL BE.

But in the fullness of time, so far, ZFS has gotten us closer than anything else to that (impossible, unacheivable) ideal.

Maybe put arming code somewhere completely different?

Of course it's possible to create more and more improbable scenarios. That's not to say those never happen; I've seen really improbable occurrences.