Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wound up causing a major outage of my services and destroying my home dir

ariadne.space

Post reply on HN
Print view View original

31–40 of 56 posts

Re: I wound up causing a major outage of my services and destroying my home dir

#31

Post by bombcar »

db48x wrote: post #3
theamk wrote: post #2

.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.

It’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?

The command told it to reattach the drive and they expected it to rebuild but instead md saw part of the mirror and simply added the drive as a good drive.

And the drive would return bad data instead of just erroring out - this is sometimes a difference between “home use” drives vs enterprise - enterprise assume you’re in a multi redundant raid setup and instead of retrying will just fail out fast and let the raid take care of it. Home drives will continue retrying and perhaps even eventually give you what they got - the idea being you’d rather have one byte bad of a word doc than lose a 4K chunk of it.

ZFS catches these tricks because the checksums will fail. Multi-redundancy RAID can do it - but since most are enterprise they are built with the assumption the drive will return good data - or none.

This can be the worst scenario as corruption can be silent and infect all backups after a time.

Re: I wound up causing a major outage of my services and destroying my home dir

#32

Post by vidarh »

hddherman wrote: post #4

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…

My home fileserver has drives from four different manufacturers after I 20 years ago had bad experiences with closely correlated drive failures (the infamous IBM Death Star batch)

I've never deployed anything without at a minimum mixing batches, and usually vendors since.

We avoided data loss, but had a tense two weeks while cycling drives out as fast as we could (we had backups, but having to shut everything down to replace drives and restore would have been massively costly)

Re: I wound up causing a major outage of my services and destroying my home dir

#33

Post by jsiepkes »

pabs3 wrote: post #24
jsiepkes wrote: post #20

Earlier quoted context omitted.

That would have consequences reaching far beyond ZFS. A blastzone which no one will be able to oversee. I would therefor say it is extremely unlikely any company (let alone Oracle) would ever sign-off on such a thing.

PS: I didn't think CDDL was used much outside of Solaris (now proprietary again and the forks of it aren't active I thought) and OpenZFS. Are there other significant CDDL licensed codebases?

> Solaris (now proprietary again and the forks of it aren't active I thought)

The Solaris forks (based on Illumos[1]) such as SmartOS[2] are pretty active!

> Are there other significant CDDL licensed codebases?

A lot of the Java stuff was also CDDL licensed. Though some of that (larger projects such as OpenJDK) were already dual licensed with "GPLv2 + classpath exception".

I also doubt it was used much outside of Sun. But even it's just the Sun stuff I still think no corporation would ever dare to make such a license update.

[1] https://www.illumos.org/ [2] https://wiki.smartos.org/

Re: I wound up causing a major outage of my services and destroying my home dir

#34

Post by hyperman1 »

One part I noticed: The backup strategy. If ariadne fucks up this bad, mere mortals have no chance.

I understand her very well, btw, as it took a minor data loss to teach me it will happen to all of us. I just got lucky.

But consider the situation as an end user: zillions of options, but no clear strategy. Simply choosing what to back up and how many times is expert level IT. Choosing a software package and storage location is headache inducing for expert level IT. Validating long term all the effort will bear fruit on the worst day of your computers life brings enterprises to their knees.

Re: I wound up causing a major outage of my services and destroying my home dir

#35

Post by vero2 »

flyinghamster wrote: post #23
hddherman wrote: post #4

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…

I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to…

> were not in their expected locations on the new box

This is why one should not use sda/sdb or vda equivalent but use /dev/disk/by-uuid/

Re: I wound up causing a major outage of my services and destroying my home dir

#36

Post by ziml77 »

YATA0 wrote: post #15

Every generation rediscovers why ZFS exists, and why it was designed the way it is.

One thing I like the idea of, that ZFS doesn't have, is redundancy broken down to a smaller size than the the whole disk so I don't need matched disk sizes and can easily expand a pool. Btrfs does this.

Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.

Re: I wound up causing a major outage of my services and destroying my home dir

#37

Post by Santosh83 »

hyperman1 wrote: post #34

One part I noticed: The backup strategy. If ariadne fucks up this bad, mere mortals have no chance. I understand her very well, btw, as it took a minor data loss to teach me it will happen to all of us. I just got lucky. But consider the situation as an end user: zillions of options, but no clear strategy. Simply choosing what to back up and how many times is expert level IT. Choosing a software package and storage l…

Non-tech end users are expected to backup everything to the Cloud. Not just backup but even store working copies. In fact non-technical users can, at most, plug in an external HDD/SSD and backup their files to it, but unless they take pains to keep a 2nd copy (and both copies in sync), it is no better than having your files on the system drive. Anything beyond this is too technical at the moment for laymen to use. I don't know why that is so. Many programs are more difficult/technical to use than they have to be. It's like some kind of unconscious gatekeeping.

Re: I wound up causing a major outage of my services and destroying my home dir

#38

Post by taneq »

hddherman wrote: post #4

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…

Slight segue but you sound like you know about ZFS (and if not, Cunningham's Law abides :D )

Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback would be annoying but acceptable.

Re: I wound up causing a major outage of my services and destroying my home dir

#39

Post by oarsinsync »

Helmut10001 wrote: post #8
theamk wrote: post #2

.. and that's how people end up with ZFS. My story was cheap SATA cables (or was it cheap power supplies? I never found out) introducing silent corruption and breaking decades of archived files.

While switching from Hardware Raid1 to ZFS recently, I had 1) sata cables failing on me and 2) a 8088-8087 PCI-Bracket corrupting half of the SATA connections through it. I would have never detected this without ZFS (scrub). All was fine for weeks. I did `fio` speedtests, `smart --long`, `dmesg` monitoring. Nothing. Until I ran the first scrub and 22 CRC read errors appeared in the 6x8TB RaidZ2. Much worse after swit…

Can you share what you use to capture the data (configs) and how you've configured your alerting?

Re: I wound up causing a major outage of my services and destroying my home dir

#40

Post by Filligree »

XorNot wrote: post #30
flyinghamster wrote: post #23

Earlier quoted context omitted.

I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to…

You were never at any risk in this scenario. The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).

Mostly true, but I've found this doesn't work if one of the drives is missing. It doesn't handle renames simultaneously with drive failure, or at least it didn't four years ago.

Using /dev/disk/by-uuid is always safe, though.

Post reply on HN