Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

Post reply on HN
Print view View original

211–220 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#211

Post by louwrentius »

rollcat wrote: post #197
snvzz wrote: post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is…

I feel that on HN people tend to be a bit pedantic about topics like data integrity, and in business settings I actually agree with them.

But for residential use, risks are just different and as you point out, you have no options except to only use a desktop workstation with ECC. People like/need laptops so that’s not realistic for most people. Just run Linux/freebsd with ZFS isn’t reasonable advice to me.

What I feel most strongly about is that it’s all about circumstances, context and risk evaluation. And I see so much blanket absolutist statements that don’t think about the reality of life and people’s circumstances.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#212

Post by bombcar »

369548684892826 wrote: post #170
AndrewDavis wrote: post #121

Earlier quoted context omitted.

I had a significant data loss years ago. I was young and only had a deskop, so all my data was there. So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good. Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young wit…

There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.

I have an MP3 file that still skips to this day, because a few frames were corrupted on disk twenty years ago.

I could probably find a new copy of it online, but that click is a good reminder about how backups aren’t just copies but have to be verified.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#213

Post by dist-epoch »

bboygravity wrote: post #133
Gud wrote: post #130

Earlier quoted context omitted.

Electronics is absolutely sensitive to this. Please use filters.

Filters won't help against prolonged periods of higher/lower voltages though.

But computer equipment uses switched power supplies which doesn't care about voltage, as long as there is enough power.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#214

Post by jll29 »

sschueller wrote: post #127
orbital-decay wrote: post #111

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

Reminds me of the time back in the day when Dell shipped us a server with drives serial numbers being consecutive. Of course both failed at the same time and I spent an all nighter doing a restore.

I ordered my NAS drives on Amazon, to avoid getting the same batch (all consecutive serial numbers) I used amazon.co.uk for one half and amazon.de for the other half of them. One could also stage the orders in time.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#215

Post by nubinetwork »

rollcat wrote: post #197
snvzz wrote: post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is…

> - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL)

> - Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel.

So modify the ZFS source so it appears as a external GPL module... just don't tell anyone or distribute it...

I can't say much about dracut or having to build the module from source... as a Gentoo user, I do it about once a month without any issues...

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#216

Post by defrost »

rollcat wrote: post #197
snvzz wrote: post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is…

Off the shelf commodity home NAS systems with ZFS onboard?

eg: https://www.qnap.com/en-au/operating-system/quts-hero

IIRC (been a while since I messed with QNAP) QuTS hero would be a modded Debian install with ZFS baked in and a web based admin dasboard.

https://old.reddit.com/r/qnap/comments/15b9a0u/qts_or_quts_h...

As a rule of thumb (IMHO) steer clear of commodity NAS Cloud add ons, such things attract ransomware hackers like flies to a tip whether it's QNAP, Synology, or InsertVendorHere.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#217

Post by freeone3000 »

rollcat wrote: post #197
snvzz wrote: post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is…

“Tainting” the kernel doesn’t affect operations, though. You’re not allowed to redistribute it with changes — but you, as an entity, can freely use ZFS and the kernel together without restriction. Linux plus zfs works fine.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#219

Post by simoncion »

rollcat wrote: post #197
snvzz wrote: post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory. While I don't disagree with this statement, consider the reality: - APFS has metadata checksums, but no data checksums. WTF Apple? - Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is…

> I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+.

I've been running btrfs as the primary filesystem for all of my desktop machines since shortly after the on-disk format stabilized and the extX->btrfs in-place converter appeared [0], and for my home servers for the past ~five years. In the first few years after I started using it on my desktop machines, I had four or five "btrfs shit the bed and trashed some of my data" incidents. I've had zero issues in the past ~ten years.

At $DAYJOB we use btrfs as the filesystem for our CI workers and have been doing so for years. Its snapshot functionality makes creating the containers for CI jobs instantaneous, and we've had zero problems with it.

I can think of a few things that might separate me from the folks who report issues that they've had within the past five-or-ten years:

* I don't use ANY of the built-in btrfs RAID stuff.

* I deploy btrfs ON TOP of LVM2 LVs, rather than using its built-in volume management stuff. [1]

* I WAS going to say "I use ECC RAM", but one of my desktop machines does not and can never have ECC RAM, so this isn't likely a factor.

The BTRFS features I use at home are snapshotting (for coherent point-in-time backups), transparent compression, the built-in CoW features, and the built-in checksumming features.

At work, we use all of those except for compression, and don't use snapshots for backup but for container volume cloning.

[0] If memory serves, this was around the time when the OCZ Vertex LE was hot, hot shit.

[1] This has actually turned out to be a really cool decision, as it has permitted me to do low- or no- downtime disk replacement or repartitioning by moving live data off of local PVs and on to PVs attached via USB or via NBD.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#220

Post by bitexploder »

userbinator wrote: post #117
bitexploder wrote: post #93

Earlier quoted context omitted.

Drives are sealed anyway. Humidity maybe. Dust can’t really get in. Power or bad batch of drives.

Except for the helium-filled ones, they aren't sealed; there is a very fine filter that equalises atmospheric pressure. (This is also why they have a maximum operating altitude --- the head needs a certain amount of atmospheric pressure to float.)

How does the helium stay in if it is not sealed? I am not familiar with hard drive construction, but helium is notoriously good at escaping.
Post reply on HN