I wound up causing a major outage of my services and destroying my home dir
1–10 of 56 posts
Re: I wound up causing a major outage of my services and destroying my home dir
#2.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.
Re: I wound up causing a major outage of my services and destroying my home dir
#3.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.
It’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?
Re: I wound up causing a major outage of my services and destroying my home dir
#4> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use.
Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well.
Anyway, ZFS is a great choice either way. Definitely implement snapshots and a regular backup habit with plain zfs send/recv or syncoid.
Re: I wound up causing a major outage of my services and destroying my home dir
#5there's so many layers that could have caught the issue, even if you don't use ZFS... What's the point of splurging on EPYC with 256 GB of RAM if you don't want to spend a penny on resilience
Re: I wound up causing a major outage of my services and destroying my home dir
#6Welcome to the club. If you haven’t taken down prod at least once you’re not trying hard enough
Re: I wound up causing a major outage of my services and destroying my home dir
#7TLDR: Lost a lot of diy hosted infra from storage problems. Conclusion: do backups.
Re: I wound up causing a major outage of my services and destroying my home dir
#8.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.
While switching from Hardware Raid1 to ZFS recently, I had 1) sata cables failing on me and 2) a 8088-8087 PCI-Bracket corrupting half of the SATA connections through it. I would have never detected this without ZFS (scrub). All was fine for weeks. I did `fio` speedtests, `smart --long`, `dmesg` monitoring. Nothing. Until I ran the first scrub and 22 CRC read errors appeared in the 6x8TB RaidZ2. Much worse after switching drives and cables around. I spend like $400 for different cable-replacements, but finally identified the failing ones and all is good since then.
Now, I am more observant. Scrub once a month. Monitoring of SSD TBW, UDMA-CRC, Temperature, Watt, Fan-Speed in InfluxDB 2.0 with alerts set. Besides, ZFS mails all the important status info.
ZFS is really good for peace of mind, resilience and predictability, not necessarily for speed (but not exclusively, either), or your wallet.
Re: I wound up causing a major outage of my services and destroying my home dir
#9.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.
It’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?
MDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code.
Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata trees to validate which copy of the data is correct and repair the corrupted copy. 3-disk makes this even more robust, as you get an exponential growth of data-metadata pairs to pull from. If any one pair matches, that data can be assumed to be correct.
Re: I wound up causing a major outage of my services and destroying my home dir
#10> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…
I agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed.
I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar product lines) helps ensure you're using SSDs that should hopefully fail at different times.