I put critical/unique data on Intel X-25E... 64GB 65nm SLC from 2011 with 100.000 writes per bit, peak SSD!
I wound up causing a major outage of my services and destroying my home dir
21–30 of 56 posts
Re: I wound up causing a major outage of my services and destroying my home dir
#22Earlier quoted context omitted.
All Oracle needs to do is release CDDL 2.0 that says "At your option, you may choose to relicense CDDL 1.0 licensed code under any of the MPLv2, LGPLv2+, GPLv2+, AGPLv3+ licenses" and then anyone can relicense any CDDL 1.0 code from any copyright holder under those licenses, since CDDL 1.0 contains a license upgrade clause. That would solve the situation for OpenZFS contributor's code too and also Oracles code and al…
That would have consequences reaching far beyond ZFS. A blastzone which no one will be able to oversee. I would therefor say it is extremely unlikely any company (let alone Oracle) would ever sign-off on such a thing.
Its a pretty simple way to achieve compatibility between CDDL and other license spheres, which can only be a good thing really, personally I consider that incompatibility to have been a mistake that didn't really help anyway. Achieving compatibility is unlikely something Oracle cares about though, so the license situation isn't going to change anyway. BTW, they have relicensed things under the GPL before (IIRC ZFS for grub and ISTR something around dtrace), so they aren't adverse to it.
Re: I wound up causing a major outage of my services and destroying my home dir
#23> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…
I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to put them back where they were, and the pool came up fine. whew
I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.
Re: I wound up causing a major outage of my services and destroying my home dir
#24Earlier quoted context omitted.
All Oracle needs to do is release CDDL 2.0 that says "At your option, you may choose to relicense CDDL 1.0 licensed code under any of the MPLv2, LGPLv2+, GPLv2+, AGPLv3+ licenses" and then anyone can relicense any CDDL 1.0 code from any copyright holder under those licenses, since CDDL 1.0 contains a license upgrade clause. That would solve the situation for OpenZFS contributor's code too and also Oracles code and al…
That would have consequences reaching far beyond ZFS. A blastzone which no one will be able to oversee. I would therefor say it is extremely unlikely any company (let alone Oracle) would ever sign-off on such a thing.
PS: I didn't think CDDL was used much outside of Solaris (now proprietary again and the forks of it aren't active I thought) and OpenZFS. Are there other significant CDDL licensed codebases?
Re: I wound up causing a major outage of my services and destroying my home dir
#25There are a lot of comments here about how the author should've used ZFS, or how well now they know about ZFS, but the post addresses this: > The other big elephant in the room, of course, is “why weren’t you using ZFS in the first place”. While it is true that Alpine has supported ZFS for years, I’ve been hesitant to use it due to the CDDL licensing. In other words, I chose the mantra instilled in me about GPL compa…
I'm pretty sure btrfs would detect these issues as well. It has certainly saved me before thanks to checksumming.
Re: I wound up causing a major outage of my services and destroying my home dir
#26You'll often find angry greybeards on here griping about how you shouldn't do X or Y. Don't run your own mail, don't write your own crypto, don't run K8s. And you'll also find greenhorns talking about how they've been doing X for 10 years and never had a problem. The thing is, both perspectives are valid. It's true that you can sometimes do really complicated things yourself and it can work out fine. And you can also…
You seem to conflate backups with off-site backups, and concluding that off-site backups are hard. However, nothing in this story would require off-site backups. A separate backup server on-site would have been fine here. Of course, off-site backups are good , but would not have been actually needed in this case.
Was going to say the same. Anyway, off-site backups aren’t that expensive, if you go with Backblaze or Wasabi. $5/TB/mo.
Re: I wound up causing a major outage of my services and destroying my home dir
#27> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…
I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to…
> I've loved seeing the progress of ZoL. Whether I use Ubuntu's prebuilt modules or Debian's DKMS-based setup, it Just Works. But I'm still leery of running Linux with a ZFS root filesystem. I've compromised and use ext4 for root and put my home directory on ZFS.
If you want to use ZFS for root, I can heartily recommend NixOS. I've never had a problem with it there. Of course there are other downsides, but there are plenty of upsides as well.
Re: I wound up causing a major outage of my services and destroying my home dir
#28Earlier quoted context omitted.
It’s pretty remarkable actually just how bad that failure mode was. How did mdadm manage to cause such havoc after the broken device was reattached?
MDADM RAID1 does not seem to have the ability to recover from silent data corruption. In the standard two-disk RAID1, it's actually impossible to recover, since it doesn't know what copy is the uncorrupted one. With three disk RAID1 it would theoretically be possible to use a majority vote, but I don't believe MDADM has such code. Btrfs raid1, on the other hand, uses the checksums of the data stored in its metadata t…
Not sure how well this works out in practice, but if one drive returns a read error for a sector and the other returns the corresponding sector without an error, it should be a simple matter to trust the latter. Of course that assumes that storage corruption occurred after uncorrupted data was written to disk.
Modern drives have rather large per physical sector checksums to detect the occurrence of these errors and presumably retry a couple times before returning an I/O error if configured correctly.
In addition a good I/O bus should have error detection checksums for data in transit, and good hardware should have ECC RAM as well.
Re: I wound up causing a major outage of my services and destroying my home dir
#29Earlier quoted context omitted.
I agree with getting SSDs from different vendors, for the following reason: I once had a FreeNAS system fail. It was using two (mirrored) SSDs to hold the OS. One SSD failed; while I was procuring a replacement, the second failed. I later found that both SSDs were from the same batch, which (I think) is why both failures happened so close together. Purchasing same-size SSDs from different vendors (with similar produc…
This was common knowledge on the internet before the "modern web" era. Never use in a RAID disks from the same vendor. When you have to use disks from the same vendor try hard to get different type/ batches. And backup.
> Never use in a RAID disks from the same vendor.
Wrong. The advice is to use the same vendor/model but from a different production run (batch).
Re: I wound up causing a major outage of my services and destroying my home dir
#30> Nonetheless, my plans are to rebuild treefort with ZFS and SSDs from another vendor. Whatever happened with the Samsung SSDs has made me anxious enough that I don’t want to trust them for continued production use. Not sure if having one bad experience with mdraid is worth switching SSD vendors as well. You can mix and match different vendors, but they may have different performance characteristics as well. Anyway,…
I first heard of ZFS back around 2009, and I don't regret my decision to go with it. But it isn't a panacea either. I nearly wiped out my storage pool one time when I moved it to a different machine without exporting it first. The pool came off an OpenIndiana system and the drives were not in their expected locations on the new box. Whoooops, the whole pool showed up FAULTED. After my panic settled down, I decided to…
You were never at any risk in this scenario.
The only thing exporting a pool does is declare "I am not being mounted by someone else". The FAULTED state would be because the search paths for the devices didn't exist. Clearing /etc/zfs/zpool.cache and doing `zpool import -f` would fix it entirely (ZFS identifies the disks by UUIDs in the headers AFAIK).