Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

171–180 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#171
post #163

Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/

That was a decision Linus regretted[1]. There has been some recent discussion about this here on Hacker News[2]. [1] https://linuxiac.com/torvalds-expresses-regret-over-merging-... [2] https://news.ycombinator.com/item?id=41407768

Context. Linux regrets it because bcachefs doesn't have same commitment to stability as Linux.

Kent wants to fix a bug with large PR

Linux doesn't want to merge and review PR that touches so many non-bcachefs things.

They're both right in a way. Kent wants bcachefs to be stable/work good, Linus wants Linux to be stable.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#172

> This NAS is very quiet for a NAS (video with audio). Big (large radius) fans can move a lot of air even at low RPM. And be much more energy efficient. Oxide Computer, in one of their presentations, talks about using 80mm fans, as they are quiet and (more importantly) don't use much power. They observed, in other servers, as much as 25% of the power went just to powering the fans, versus the ~1% of theirs: * https:/…

Interesting - I'm used to desktop/workstation hardware where 80mm is the smallest standard fan (aside from 40mm's in the near-extinct Flex ATX PSU), and even that is kind of rare. Mostly you see 120mm or 140mm.

Yeah. In a home environment you should absolutely use desktop gear. I have 5 80mm and one 120mm PWM fans in my NAS and they are essentially silent as they can't be heard over the sound of the drives (which is essentially the noise floor for a NAS).

It is necessary to use good PWM fans though if concerned about noise as cheaper ones can "tick" annoyingly. Two brands I know to be good in this respect are Be Quiet! and Noctua. DC would in theory be better but most motherboards don't support it (would require an external controller and thermal sensors I think).

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#173
post #148

Earlier quoted context omitted.

> No serious person designing a filesystem today would say it's okay to misplace your data. Former LimeWire developer here... the LimeWire splash screen at startup was due to experiences with silent data corruption. We got some impossible bug reports, so we created a stub executable that would show a splash screen while computing the SHA-1 checksums of the actual application DLLs and JARs. Once everything checked out…

It sounds like a fun comp sci exercise to optimise the algo for randomised block download to reduce disk operations but maintain resilience. Presumably it would vary significantly by disk cache sizes. It's not my field, but my impression is that it would be equally resilient to just randomise the start block (adjust spacing of start blocks according to user bandwidth?) then let users just run through the download ser…

For the completely randomized algorithm, my initial prototype was to always download the first block if available. After that, if fewer than 4 extents (continuous ranges of available bytes) were downloaded locally, randomly chose any available block. (So, we first get the initial block, and 3 random blocks.) If 4 or more extents were available locally, then always try the block after the last downloaded block, if available. (This is to minimize disk seeks.) If the next block isn't available, then the first fallback was to check the list of available blocks against the list of next blocks for all extents available locally, and randomly choose one of those. (This is to chose a block that hopefully can be the start of a bunch of sequential downloads, again minimizing disk seeks.) If the first fallback wasn't available, then the second fallback was to compute the same thing, except for the blocks before the locally available extents rather than the blocks after. (This is to avoid increasing the number of locally available extents if possible.) If the second fallback wasn't available, then the final fallback was to randomly uniformly pick one of the available blocks.

Trying to extend locally available extents if possible was desirable because peers advertised block availability as pairs of , so minimizing the number of extents minimized network message sizes.

This initial prototype algorithm (1) minimized disk seeks (after the initial phase of getting the first block and 3 other random blocks) by always downloading the block after the previous download, if possible. (2) Minimized network message size for advertising available extents by extending existing extents if possible.

Unfortunately, in simulation this initial prototype algorithm biased availability of blocks in rare files, biasing in favor of blocks toward the end of the file. Any bias is bad for rapidly spreading rare content, and bias in favor of the end of the file is particularly bad for audio and video file types where people like to start listening/watching while the file is still being downloaded.

Instead, the algorithm in the initial production implementation was to first check the file extension against a list of extensions likely to be accessed by the user while still downloading (mp3, ogg, mpeg, avi, wma, asf, etc.).

For the case where the file extension indicates the user is unlikely to access the content until the download is finished (the general case algorithm), look at the number of extents (continuous ranges of bytes the user already has). If the number of extents is less than 4, pick any block randomly from the list of blocks that peers were offering for download. If there are 4 or more extents available locally, for each end of each extent available locally, check the block before it and the block after it to see if they're available for download from peers. If this list of available adjacent blocks is non-empty, then randomly chose one of those adjacent blocks for download. If the list of available adjacent blocks is empty, then uniformly randomly chose from one of the blocks available from peers.

In the case of file types likely to be viewed while being downloaded, it would download from the front of the file until the download was 50% complete, and then randomly either download the first needed block, or else use the previously described algorithm, with the probability of using the previous (randomized) algorithm increasing as the percentage of the download completed increased. There was also some logic to get the last few chunks of files very early in the download for file formats that required information from a file footer in order to start using them (IIRC, ASF and/or WMA relied on footer information to start playing).

Internally, there was also logic to check if a chunk was corrupted (using a Merkle tree using the Tiger hash algorithm). We would ignore the corrupted chunks when calculating the percentage completed, but would remove corrupted chunks from the list of blocks we needed to download, unless such removal resulted in an empty list of blocks needed for download. In this way, we would avoid re-downloading corrupted blocks unless we had nothing else to do. This would avoid the case where one peer had a corrupted block and we just kept re-requesting the same corrupted block from the peer as soon as we detected corruption. There was some logic to alert the user if too many corrupted blocks were detected and give the user options to stop the download early and delete it, or else to keep downloading it and just live with a corrupted file. I felt there should have been a third option to keep downloading until a full-but-corrupt download was had, retry downloading every corrupt block once, and then re-prompt the user if the file was still corrupt. However, this option would have resulted in more wasted bandwidth and likely resulted in more user frustration due to some of them hitting "keep trying" repeatedly instead of just giving up as soon as it was statistically unlikely they were going to get a non-corrupted download. Indefinite retries without prompting the user were a non-starter due to the amount of bandwidth they would waste.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#174
post #159

Earlier quoted context omitted.

Or in my case, a key filesystem metadata block that ruins everything. :s

I only know about FAT but these "key file metadata blocks" are redundant, so you need really special double-plus bad luck to do that.

so I can consider myself very lucky and unlucky at the same time. I had data corruption on zfs filesystem that destroyed whole pool to unrecoverable state (zfs was segfaulting while trying to import, all recovery zfs features where crashing zfs module and required reboot) the lucky part is that this happened just after (something like next day) I migrated whole pool to another (bigger) server/pool so that system was already scheduled for full disk wipe

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#175
post #159

Earlier quoted context omitted.

Or in my case, a key filesystem metadata block that ruins everything. :s

I only know about FAT but these "key file metadata blocks" are redundant, so you need really special double-plus bad luck to do that.

It was ext4, and I’ve had it happen two different times - in fact, I’ve never had it happen in a ‘good’ recoverable way before that I’ve ever seen.

It triggered a kernel panic in every machine that I mounted it in, and it wasn’t a media issue either. Doing a block level read of the media had zero issues and consistently returned the exact same data the 10 times I did it.

Notably, I had the same thing happen using btrfs due to power issues on a Raspberry Pi (partially corrupted writes resulting in a completely unrecoverable filesystem, despite it being in 2x redundancy mode).

Should it be impossible? Yes. Did it definitely, 100% for sure happen? You bet.

I never actually lost data on ZFS, and I’ve done some terrible things to pools before that took quite awhile to unbork, including running it under heavy write load with a machine with known RAM problems and no ECC.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#176

Earlier quoted context omitted.

I had a significant data loss years ago. I was young and only had a deskop, so all my data was there. So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good. Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young wit…

There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.

It happens all the time. Have a plan, perform fire drills. It's a lot of time and money, but there's no equivalent feeling to unfucking yourself quite like being able to get your lost, fragile data back.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#177
post #59

In my experience the environment where the drives are running makes a huge difference in longevity. There's a ton more variability in residential contexts than in data center (or even office) space. Potential temperature and humidity variability is a notable challenge but what surprised me was the marked effect of even small amounts of dust. Many years ago I was running an 8x500G array in an old Dell server in my bas…

It is most likely the model's fault. I once had a machine with 36 Seagate ST3000DM001, they were failing almost once a month -- see the annual failure rate here https://www.backblaze.com/blog/best-hard-drive-q4-2014/

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#178
post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

Most people don’t run ZFS on their laptop, desktop so let’s not pretend it’s such a huge deal.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#179

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

I bought the drives in several batches from 2 or 3 different shops.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#180
post #129

> Losing the system due to power shenanigans is a risk I accept. There is another (very rare) failure an ups protects against, and that's imbalance in the electricity. You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house. First job I worked at had multiple servers die like that,…

True, this is also what I mean with power shenanigans.

My server is off most off the time, disconnected. But even if it wasn’t, I just accept the risk.

Post reply on HN