Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wound up causing a major outage of my services and destroying my home dir

ariadne.space

Post reply on HN
Print view View original

41–50 of 56 posts

Re: I wound up causing a major outage of my services and destroying my home dir

#41

Post by matja »

vero2 wrote: post #35
flyinghamster wrote: post #23

Earlier quoted context omitted.

I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to…

> were not in their expected locations on the new box This is why one should not use sda/sdb or vda equivalent but use /dev/disk/by-uuid/

Can also use cachefile=none and the name does not matter at all. Every disk vdev has a unique guid in the ZFS label area. Without the pool cachefile the guid is used to reliably assemble the pool, instead of the kernel device path.

Re: I wound up causing a major outage of my services and destroying my home dir

#42

Post by wheybags »

hulitu wrote: post #12
CaliforniaKarl wrote: post #10

Earlier quoted context omitted.

I agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed. I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar produc…

This was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.

I very intentionally bought two Samsung ssd from different countries, so I would get different batches. They show up on my door. Same batch, only a few hundreds apart in their serial number :(

Re: I wound up causing a major outage of my services and destroying my home dir

#43

Post by hddherman »

taneq wrote: post #38
hddherman wrote: post #4

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…

Slight segue but you sound like you know about ZFS (and if not, Cunningham's Law abides :D ) Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback wo…

Disclaimer: I'm a bumbling idiot who does this type of thing as a hobby.

Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place.

In your scenario, I understand that the original physical host where you're making a backup of a VM has no redundancy or data integrity guarantees? I'd just like to point out that backups are good, but if all you have are backups of the same corrupted data, then they are pretty much useless.

My approach is to have redundancy and data integrity guarantees (ZFS mirror) at the source and have offline backups on external hard drives. The original data is likely to be OK, and well, if I need to recover from a backup, I'll be happy enough with what I have there anyway.

Re: I wound up causing a major outage of my services and destroying my home dir

#44

Post by hatware »

throwaway984393 wrote: post #14

You'll often find angry greybeards on here griping about how you shouldn't do X or Y. Don't run your own mail, don't write your own crypto, don't run K8s. And you'll also find greenhorns talking about how they've been doing X for 10 years and never had a problem. The thing is, both perspectives are valid. It's true that you can sometimes do really complicated things yourself and it can work out fine. And you can also…

> don't write your own crypto

I'm no greybeard, but this one IS important.

Re: I wound up causing a major outage of my services and destroying my home dir

#45

Post by jewel »

christophilus wrote: post #26
teddyh wrote: post #18

Earlier quoted context omitted.

You seem to conflate backups with off-site backups, and concluding that off-site backups are hard. However, nothing in this story would require off-site backups. A separate backup server on-site would have been fine here. Of course, off-site backups are good , but would not have been actually needed in this case.

Was going to say the same. Anyway, off-site backups aren’t that expensive, if you go with Backblaze or Wasabi. $5/TB/mo.

If you have a lot of data then a cheaper choice would be to put a raspberry pi at your mom's house, add a few external drives, forward a port, and send backups with restic.

Re: I wound up causing a major outage of my services and destroying my home dir

#46

Post by mst »

vidarh wrote: post #32
hddherman wrote: post #4

> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…

My home fileserver has drives from four different manufacturers after I 20 years ago had bad experiences with closely correlated drive failures (the infamous IBM Death Star batch) I've never deployed anything without at a minimum mixing batches, and usually vendors since. We avoided data loss, but had a tense two weeks while cycling drives out as fast as we could (we had backups, but having to shut everything down to…

Those were actually some of my favourite drives cost wise ... -after- the problems happened.

Turned out at least on the ones I had that it was always the outer edges of the platter that failed, so a (from 20 year old memory, may be wrong) 60Gb Death Star if carefully partitioned out to a maximum of about 58Gb ran absolutely beautifully and you could get them dirt cheap because (understandably if you weren't a crazy person like me) almost nobody wanted to touch them at that point.

Re: I wound up causing a major outage of my services and destroying my home dir

#47

Post by mst »

hddherman wrote: post #43
taneq wrote: post #38

Earlier quoted context omitted.

Slight segue but you sound like you know about ZFS (and if not, Cunningham's Law abides :D ) Does a VM on one physical host, backed up via nightly disk image on a geographically separate physical host, with the latter storing the backups on a 2-disk mirrored ZFS pool, count as a reasonable 3-2-1 backup regime? It's not storing mission critical up-to-the-minute user data, 'just' company files, so a one-day rollback wo…

Disclaimer: I'm a bumbling idiot who does this type of thing as a hobby. Add nightly or weekly offline backups and you should be good to go. And by offline, I mean physically connecting a storage device, making the backup (preferrably via an automated script), and then disconnecting the device and storing it elsewhere in a safe place. In your scenario, I understand that the original physical host where you're making…

Caveat that you probably already know but others reading may not: 'zfs send' while absolutely awesome when used for things it's designed for will stream the file system byte for byte so if there's corruption brewing you will absolutely transfer the corruption along with everything else.

(taking a snapshot, mounting the snapshot, and then backing up from -that- via something 'cp -a' like is probably the safest option, IMO, though it's always possible I've missed something here in which case I hope somebody will reply correcting me)

Re: I wound up causing a major outage of my services and destroying my home dir

#48

Post by lostlogin »

wheybags wrote: post #42
hulitu wrote: post #12

Earlier quoted context omitted.

This was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.

I very intentionally bought two Samsung ssd from different countries, so I would get different batches. They show up on my door. Same batch, only a few hundreds apart in their serial number :(

If you had tried to get two that close in production, I think you’d have struggled. I didn’t bother and got drives off the stack and bought both, and they are a long way apart.

Re: I wound up causing a major outage of my services and destroying my home dir

#49

Post by halo37253 »

Honestly I hate builds like this.

I'd rock ESXI as hypervisor, and Truenas as the Storage environment.

Passthrough the SSD's and any Disk controller to TrueNas. I always recommend HDD Array to backup your primary SSD array. This could have all been avoided with ZFS, along with future worry of bitrot.

Then you can run your Primary Linux VM for Docker, and another other standalone Linux VM's you may need. You'll want to keep Truenas on a datastore connected to ESXI, but the rest of the storage back be kept on Truenas and fed back into esxi.

Not only keeps your system portable for future upgrade paths, but keeps your system reliable.

Opensource is not always the best route to take.

Re: I wound up causing a major outage of my services and destroying my home dir

#50

Post by magicalhippo »

ziml77 wrote: post #36
YATA0 wrote: post #15

Every generation rediscovers why ZFS exists, and why it was designed the way it is.

One thing I like the idea of, that ZFS doesn't have, is redundancy broken down to a smaller size than the the whole disk so I don't need matched disk sizes and can easily expand a pool. Btrfs does this. Difficulty aside, I'm wondering if it would be possible for ZFS to implement that as an option without giving up other features.

You can do it yourself at the cost of (a lot of) performance. Make say 4TB partitions on each drive, and make vdevs across the partitions.

Then you can upgrade by putting the partitions on different drives etc. Of course must then be careful not to put two partitions of the same vdev on the same disk.

The major downside is that ZFS isn't aware that the underlying devices are the same, so IO which looks nicely scheduled as far as ZFS is concerned might cause massive random IO on the underlying disks.

That said, if one could tell ZFS about which zfs-devices share physical hardware, so the IO scheduler would be aware, then that might be a viable strategy. Might be handy in case you have multiple iSCSIs sharing a 10G link as well.

Post reply on HN