Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wound up causing a major outage of my services and destroying my home dir

ariadne.space

Post reply on HN
Print view View original

51–56 of 56 posts

Re: I wound up causing a major outage of my services and destroying my home dir

#51

Post by ziml77 »

magicalhippo wrote: post #50
ziml77 wrote: post #36

Earlier quoted context omitted.

One thing I like the idea of, that ZFS doesn't have, is redundancy broken down to a smaller size than the the whole disk so I don't need matched disk sizes and can easily expand a pool. Btrfs does this. Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.

You can do it yourself at the cost of (a lot of) performance. Make say 4TB partitions on each drive, and make vdevs across the partitions. Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk. The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely schedul…

Huh. I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive. Though I guess it does make sense that you can, otherwise you wouldn't be able to use a drive participating in a vdev as a boot device.

I don't need the feature that badly though. It's just something that's interested me about Unraid (and most consumer NAS devices actually), especially since I'm running 4 2TB drives and 2 3TB drives as a single RAIDZ2 vdev in my home TrueNAS setup.

Re: I wound up causing a major outage of my services and destroying my home dir

#52

Post by halo37253 »

kova12 wrote: post #5

there's so many layers that could have caught the issue, even if you don't use ZFS... What's the point of splurging on EPYC with 256 GB of RAM if you don't want to spend a penny on resilience

It smells of Web Dev doing Web Dev things.

It works now, it'll be fine.

Sure a Docker or VM can be easy to rebuild from scratch. But a Database is not.

I learned years ago when a failed HDD took 2TB of data I will never get back.

Re: I wound up causing a major outage of my services and destroying my home dir

#53

Post by magicalhippo »

ziml77 wrote: post #51
magicalhippo wrote: post #50

Earlier quoted context omitted.

You can do it yourself at the cost of (a lot of) performance. Make say 4TB partitions on each drive, and make vdevs across the partitions. Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk. The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely schedul…

Huh. I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive. Though I guess it does make sense that you can, otherwise you wouldn't be able to use a drive participating in a vdev as a boot device. I don't need the feature that badly though. It's just something that's interested me about Unraid (and most consumer NAS devices actually), especially since…

> I hadn't even considered that it would be possible to assign partitions to the vdevs instead of the entirety of the drive.

You can even use files[1]! Mostly available for testing though.

It would be nice with a bit more flexibility with regards to the underlying storage, hence this concept. I tested it in a VM and it works fine, just slow when the underlying disks gets trashed. Mostly noticeable on heavy writes and scrubs.

Should work fine on SSDs though, and might possibly even be a boon to very fast NVMe drives (increasing concurrency).

[1]: https://openzfs.github.io/openzfs-docs/man/7/zpoolconcepts.7...

Re: I wound up causing a major outage of my services and destroying my home dir

#54

Post by db48x »

bombcar wrote: post #31
db48x wrote: post #3

Earlier quoted context omitted.

It’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?

The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive. And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid ta…

> The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.

> ZFS catches these tricks because the checksums will fail.

Yea, I think this is the right explanation. I guess it simply doesn’t keep any consistency information that it can check, whereas ZFS keeps a checksum of every block. Having used ZFS, it is difficult for me to understand how anyone could trust anything else. I don’t think that there is anything a ZFS user could do to accidentally destroy their data (aside from obvious things such as deleting the wrong filesystem or pool, which could happen in any case).

Re: I wound up causing a major outage of my services and destroying my home dir

#55

Post by zootboy »

butlerm wrote: post #28
zootboy wrote: post #9

Earlier quoted context omitted.

MDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code. Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata t…

Not sure how well this works out in practice, but if one drive returns a read error for a sector and the other returns the corresponding sector without an error, it should be a simple matter to trust the latter. Of course that assumes that storage corruption occurred after uncorrupted data was written to disk. Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and p…

Indeed. That's why I specified /silent/ corruption. If the stack returns an explicit error, that is correctable (assuming the other disks don't error and return the correct data).

Sadly, a large number of SSDs (and nearly all cheap USB thumb drives / SD cards / eMMC devices) will happily return silently corrupted data.

Re: I wound up causing a major outage of my services and destroying my home dir

#56

Post by Helmut10001 »

oarsinsync wrote: post #39
Helmut10001 wrote: post #8

Earlier quoted context omitted.

While switching from Hardware Raid1 to ZFS recently, I had 1) sata cables failing on me and 2) a 8088-8087 PCI-Bracket corrupting half of the SATA connections through it. I would have never detected this without ZFS (scrub). All was fine for weeks. I did `fio` speedtests, `smart --long`, `dmesg` monitoring. Nothing. Until I ran the first scrub and 22 CRC read errors appeared in the 6x8TB RaidZ2. Much worse after swit…

Can you share what you use to capture the data (configs) and how you've configured your alerting?

I have written somee of it down here: https://du.nkel.dev/blog/2021-05-05_proxmox_influxdb/

Currently away, will update later.

Post reply on HN