Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

Post reply on HN
Print view View original

161–170 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#161

Post by Too »

zvr wrote: post #154
Too wrote: post #128

Earlier quoted context omitted.

The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues…

Was there any change in the license that made you believe that it should be less than a issue? Or do you think people simply stopped paying attention?

Canonical took a team of lawyers to deeply review the license in 2016. It's beyond my legal skills to say if the conclusion made it more or less of an issue, at least the boundaries should now be more clear, for those who understand these matters more.

https://canonical.com/blog/zfs-licensing-and-linux

https://softwarefreedom.org/resources/2016/linux-kernel-cddl...

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#162

Post by olavgg »

manuel_w wrote: post #160

Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/

It is marked experimental, and since it was merged into the kernel there have been a few major issues that has been resolved. I wouldn't risk production data on it, but for a home lab it could be fine. But you need to ask yourself, how much time are you willing to spend if something should go wrong? I have also been running ZFS for 15+ years, and I've seen a lot of crap because of bad hardware. But with good enterprise hardware it has been working flawless.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#163

Post by clan »

manuel_w wrote: post #160

Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/

That was a decision Linus regretted[1]. There has been some recent discussion about this here on Hacker News[2].

[1] https://linuxiac.com/torvalds-expresses-regret-over-merging-...

[2] https://news.ycombinator.com/item?id=41407768

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#164

Post by pbhjpbhj »

lazide wrote: post #159
nolok wrote: post #147

Earlier quoted context omitted.

That's all fine and good until that one random lone broken block stops you from opening that file you really need.

Or in my case, a key filesystem metadata block that ruins everything. :s

I only know about FAT but these "key file metadata blocks" are redundant, so you need really special double-plus bad luck to do that.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#165

Post by JonChesterfield »

ffsm8 wrote: post #129

> Losing the system due to power shenanigans is a risk I accept. There is another (very rare) failure an ups protects against, and that's imbalance in the electricity. You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house. First job I worked at had multiple servers die like that,…

Lightning took out a modem and some nearby hardware here about a week ago. Residential. The distribution of dead vs damaged vs nominally unharmed hardware points very directly at the copper wire carrying vdsl. Modem was connected via ethernet to everything else.

I think the proper fix for that is probably to convert to optical, run along a fibre for a bit, then convert back. It seems likely that electricity will take a different route in preference to the glass. That turns out to be disproportionately annoying to spec (not a networking guy, gave up after an hour trying to distinguish products) so I've put a wifi bridge between the vdsl modem and everything else. Hopefully that's the failure mode contained for the next storm.

Mainly posting because I have a ZFS array that was wired to the same modem as everything else. It seems to have survived the experience but that seems like luck.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#166

Post by pbhjpbhj »

KMag wrote: post #148
mustache_kimono wrote: post #105

Earlier quoted context omitted.

> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. No serious person designing a filesystem today would say it's okay to misplace your data. Not long ago, on this forum, someone told me that ZFS is only good because it had no competitors in its sp…

> No serious person designing a filesystem today would say it's okay to misplace your data. Former LimeWire developer here... the LimeWire splash screen at startup was due to experiences with silent data corruption. We got some impossible bug reports, so we created a stub executable that would show a splash screen while computing the SHA-1 checksums of the actual application DLLs and JARs. Once everything checked out…

It sounds like a fun comp sci exercise to optimise the algo for randomised block download to reduce disk operations but maintain resilience. Presumably it would vary significantly by disk cache sizes.

It's not my field, but my impression is that it would be equally resilient to just randomise the start block (adjust spacing of start blocks according to user bandwidth?) then let users just run through the download serially; maybe stopping when they hit blocks that have multiple sources and then skipping to a new start block?

It's kinda mindbogglingly to me too think of all the processes that go into a 'simple' torrent download at the logical level.

If AIs get good enough before I die then asking it to create simulations on silly things like this will probably keep me happy for all my spare time!

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#167

Post by KMag »

doublepg23 wrote: post #152
KMag wrote: post #150

Earlier quoted context omitted.

How are the memory overheads of ZFS these days? In the old days, I remember balking at the extra memory required to run ZFS on the little ARM board I was using for a NAS.

That was always FUD more or less. ZFS uses RAM as its primary cache…like every other filesystem, so it if you have very little RAM for caching the performance will degrade…like every other filesystem.

But if you have a single board computer with 1 GB of RAM and several TB of ZFS, will it just be slow, or actually not run? Granted, my use case was abnormal, and I was evaluating in the early days when there were both license and quality concerns with ZFS on Linux. However, my understanding at the time was that it wouldn't actually work to have several TB in a ZFS pool with 1 GB of RAM.

My understanding is that ZFS has its own cache apart from the page cache, and the minimum cache size scales with the storage size. Did I misundertand/is my information outdated?

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#168

Post by eru »

manuel_w wrote: post #160

Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/

I'm using it. It's been ok so far, but you should have all your data backed up anyway, just in case.

I'm trying a combination where I have an SSD (of about 2TiB) in front of a big hard drive (about 8 TiB) and using the SSD as a cache.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#169

Post by nuancebydefault »

danw1979 wrote: post #139
ffsm8 wrote: post #129

> Losing the system due to power shenanigans is a risk I accept. There is another (very rare) failure an ups protects against, and that's imbalance in the electricity. You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house. First job I worked at had multiple servers die like that,…

I’ve had firsthand experience of a lightning strike hitting some gear that I maintained… My parent’s house got hit right on the TV antenna, which was connected via coax down the the booster/splitter unit in comms cupboard … then somehow it got onto the nearby network patch panel and fried every wired ethernet controller attached to the network, including those built into switch ports, APs, etc. In the network switch,…

Well this is an other order of magnitude than 'spikes on the net'. The electrical field is so intense that current will easily cross large air gaps.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#170

Post by 369548684892826 »

AndrewDavis wrote: post #121
naming_the_user wrote: post #100

Earlier quoted context omitted.

I agree. I think that the author may not have experienced these sorts of errors before. Yes, the average person may not care about experiencing a couple of bit flips per year and losing the odd pixel or block of a JPEG, but they will care if some cable somewhere or transfer or bad RAM chip or whatever else manages to destroy a significant amount of data before they notice it.

I had a significant data loss years ago. I was young and only had a deskop, so all my data was there. So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good. Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young wit…

There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.
Post reply on HN