221–230 of 237 posts
Can people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actually one of the few that can still be safer if you don't run with ECC memory. Source: Matthew Ahrens himself: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
1. ZFS doesn't come with any disk repair tools and the ones that exist are not nearly as capable as for other filesystems (the ZFS motto is that it is too costly to repair filesystems, just recover from tape instead (here we can sense the intended audience of ZFS)). If the wrong bit get's flipped your entire pool might be gone (you can of course spend months of your spare time to debug it yourself if you want to). This is not the case (to the same extent) for FAT, NTFS or EXT.
2. The more you use memory the more likely you are about to get hit. I'd argue that ZFS is a quite resource heavy filesystem and is thus more likely to actually attract bit flips. This is similar as to when using an encrypted filesystem on an overclocked CPU. There is nothing inherently more risky with encrypting your filesystem with an overclocked CPU - but overclocking your CPU increases the risk for miscalculations. And enabling encryption increases the CPU usage when accessing the filesystem by several orders of magnitude. So, in practice you quickly notice how filesystem data on encrypted drives get corrupted but not on regular drives on an ever so slightly too overclocked machine.
So, if you care about your filesystem, then yes - saying that ZFS needs ECC is quite sensible. (if you care about your data you should have backups regardless)
Earlier quoted context omitted.
So, I've sent an int32 representing a payment amount. One of the low order bits gets flipped. Can you explain to me how I'd validate it?
Compare the cost of the transaction to the price paid. First parse the payment amount from the client and store in in a bad region of memory. Then compare that variable with the transaction amount. When you read from the variable it will be corrupt and not match. Log the Error as High Priority because it shouldn't occur. Just saved you a ton with a simple if statement.
The cost of the transaction is, by definition, the price paid. `if x != x: raise_error()` only works if x is NaN.
Can people here please stop posting that ZFS needs ECC memory. Every filesystem, with any name like FAT, NTFS, EXT4 runs more safe with ECC memory. ZFS is actually one of the few that can still be safer if you don't run with ECC memory. Source: Matthew Ahrens himself: https://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=...
No. ZFS is in much greater need for ECC than most other filesystems. 1. ZFS doesn't come with any disk repair tools and the ones that exist are not nearly as capable as for other filesystems (the ZFS motto is that it is too costly to repair filesystems, just recover from tape instead (here we can sense the intended audience of ZFS)). If the wrong bit get's flipped your entire pool might be gone (you can of course spe…
Regardless of ZFS, this is why we architect our storage to cope with such potential borkage (which has, incidentially, only happened to me _ONCE_ in ~8 years of zfs in prod, had nothing to do with silent md corruption in ram and had everything to do with a nasty hw sas hba bug) -- If losing a single storage node/pool causes you problems then you're "doing it wrong" (sorry) and it makes no difference if you're using XFS, ZFS, BTRFS or whatever else...
So... I'm not sure about the ECC stuff, to me ECC really matters not at all for the simple reason that any significant deployment is using ECC anyway: even deploying a cheapo 10k pair of jbods and a tiny 1U head to run NFS or something, you'll be unlikely to even have the option of non ECC from whoever you're buying the kit from (dell? hp? bla) right?
Yes, it might work without it. It might work better somehow with it.. What does it matter when even the cheap gear comes with it anyway?
I've built a reasonable slew of ZFS backed storage (well, a 10 PB prod or so anyway, nothing compared to what some of the folks who comment here have done) and besides some hardware compat issues if you're building storage this is currently your best option to back your objstore/dfs.
ZFS as the storage backend for your DC/Cloud? Good pick. ZFS as the 'local' FS's in your VMs? I wouldn't bother, unless you need some features (it works pretty well with docker, as it goes, but I prefer to run apps on plain ext backed by zvols instead of 'zfs-on-zfs')....
Earlier quoted context omitted.
Compare the cost of the transaction to the price paid. First parse the payment amount from the client and store in in a bad region of memory. Then compare that variable with the transaction amount. When you read from the variable it will be corrupt and not match. Log the Error as High Priority because it shouldn't occur. Just saved you a ton with a simple if statement.
> Compare the cost of the transaction to the price paid. The cost of the transaction is, by definition, the price paid. `if x != x: raise_error()` only works if x is NaN.
This is a ridiculous conversation because data corruption could happen in the CPU cache, the QPI, or a number of micro-components in between the ram and cpu that could cause errors that ECC Ram can't fix. ECC ram is not a catch all for poor programming and poor validation checking period.
Earlier quoted context omitted.
I hotswap drives all the time, it's not a problem and makes a harddrive swap a 30 second task instead of a 10 minute task and doesn't incur downtime either.
Most consumer hard drives (and indeed bays) are not designed for hotswapping and it can cause damage (though maybe modern build quality is good enough that you'd be lucky most of the time). "Downtime" on your home server in your closet is a minor inconvenience at worst.
Earlier quoted context omitted.
I have hit soft errors in every desktop machine that used ECC. Either I have bad luck, ECC causes the errors or third thing. I think ECC should be mandated for anything except toys and video players.
> I have hit soft errors in every desktop machine that used ECC. Not sure if I should start getting nervous or just your RAM sucks ;) I get ECC errors only if I overclock too much , and I run the RAM overclocked all time. It's actually one of the reasons I wanted ECC.
Earlier quoted context omitted.
What about memory-error corrupted application data (or application logic) where the corruption occurred on load balancers or web application servers? There's more to data integrity than security holes.
If you write code that detects stack smashing and illegal dereferences then you can terminate the webservice and either have a watchdog restart it or if it crashes multiple times have it taken out of service by the load balancer. There are plenty of ways to handle hardware errors without throwing out the hardware and getting "better" hardware. Technically, You could have a faulty component somewhere between the Ram a…
Earlier quoted context omitted.
What about memory-error corrupted application data (or application logic) where the corruption occurred on load balancers or web application servers? There's more to data integrity than security holes.
If you write code that detects stack smashing and illegal dereferences then you can terminate the webservice and either have a watchdog restart it or if it crashes multiple times have it taken out of service by the load balancer. There are plenty of ways to handle hardware errors without throwing out the hardware and getting "better" hardware. Technically, You could have a faulty component somewhere between the Ram a…
Regarding faulty components, that is one part of ECC's job, but the other part is correcting the regular bit flips that happen with nominally operating DRAM.
Flagging faulty components is more useful than you propose. There are not that many places where this corruption can occur, so being able to rule out RAM is very useful. The example you used, CPU caches, is actually already covered by ECC in most CPUs, including reasonably recent x86/amd64.
The tradeoff would be more worthy of thought if ECC was much more expensive
Earlier quoted context omitted.
I use NAS Harddrives which are built for hotswapping. I have no idea why anybody would use a consumer harddrive in a RAID Array, the price difference is 10€ at best AFAIK.
For a home server what's the benefit you're paying for though? I don't need max performance (I use RAID for redundancy rather than anything else), and a little downtime when I replace a disk isn't an issue.
Lastly, NAS Drives have a much lower error rate than Desktop drives due to the usage of higher quality heads that increase error resistance and lifetime.