Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

Post reply on HN
Print view View original

281–290 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#281

Post by chiph »

throw0101c wrote: post #50

> This NAS is very quiet for a NAS (video with audio). Big (large radius) fans can move a lot of air even at low RPM. And be much more energy efficient. Oxide Computer, in one of their presentations, talks about using 80mm fans, as they are quiet and (more importantly) don't use much power. They observed, in other servers, as much as 25% of the power went just to powering the fans, versus the ~1% of theirs: * https:/…

My Synology uses two 120mm fans and you can barely hear them (it's on the desk next to me). I'm sold on the idea of moving more volume at less speed.

(which I understand can't happen in a 1U or 2U chassis)

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#282

Post by DistractionRect »

manuel_w wrote: post #160

Discussions on checksumming filesystems usually revolve around ZFS and BTRFS, but has someone any experience with bcachefs? It's upstreamed in the linux kernel, I learned, and is supposed to have full checksumming. The author also seems to take filesystem responsibility seriously. Is anyone using it around here? https://bcachefs.org/

I'm optimistic about it, but probably won't switch over my home lab for a while. I've had quirks with my (now legacy) zsys + zfs on root for Ubuntu, but since it's a common config//widely used for years it's pretty easy to find support.

I probably won't use bcachefs until a similar level of adoption/community support exists.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#283

Post by layer8 »

Tinned_Tuna wrote: post #269
orbital-decay wrote: post #111

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

I've seen this happen to a friend. Back in the noughties they built a home NAS similar to the one in the article, using fewer (smaller) drives. It was in RAID5 configuration. It lasted until one drive died and a second followed it during the rebuild. Granted, it wasn't using ZFS, there was no regular scrubbing, 00s drive failure rates were probably different, and they didn't power it down when not using it. The point…

This is the reason why I would always use RAID 6. A second drive failing during rebuild is significantly likely.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#284

Post by Fastjur »

leptons wrote: post #275
Fastjur wrote: post #271

Earlier quoted context omitted.

I’m guessing that the 71 TiB is mostly used for media, as in, plex/jellyfin media, which is sad to loose but not unrecoverable. How would one ever store that much of personal data? I hope they have an off site backup for the all important unrecoverable data like family photos and whatnot.

I have about about 80TB (my wife's data and mine) backed up to LTO5 tape. It's pretty cheap to get refurb tape drives on ebay. I pay about $5.00/TB for tape storage, not including the ~$200 for the LTO drive and an HBA card, so it was pretty economical.

Wow that is surprisingly cheap actually.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#285

Post by prepend »

lvl155 wrote: post #249
prepend wrote: post #235

I wish he had talked more about his movie collection. I’m interested in the methods of selecting initial items as well as ones that survive in the collection for 10+ years.

Off topic but why do people run Plex and movies locally in 2024? Is “iTunes” that bad?

I don’t know about other people but I run Plex because it lets me run my home movie collection through multiple clients (Apple TV, browser, phones, etc). iTunes works great with content bought from Apple, but is useless when playing your own media files from other sources.

I just want every Disney movie available so my kids can watch without bothering me.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#286

Post by leptons »

Fastjur wrote: post #284
leptons wrote: post #275

Earlier quoted context omitted.

I have about about 80TB (my wife's data and mine) backed up to LTO5 tape. It's pretty cheap to get refurb tape drives on ebay. I pay about $5.00/TB for tape storage, not including the ~$200 for the LTO drive and an HBA card, so it was pretty economical.

Wow that is surprisingly cheap actually.

I get email alerts from ebay for anything related to LTO-5, and I only buy the cheap used tapes. They are still fine, most of them are very low use, the tapes actually have a chip in them that stores usage data like a car's odometer, so you can know how much a tape has been used. So far I trust tapes more than I'd trust a refurb hard drive for backups. And I also really like that the tapes have a write-protect notch on them, so once I write my backup data, there's no risk of having a tape accidentally get erased, unlike if I plugged in a hard drive and maybe there's some ransomware virus that automatically fucks with any hard drive drive that gets plugged in. It's just one less thing to worry about.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#287

Post by orbital-decay »

mdaniel wrote: post #276
iforgotpassword wrote: post #240

Earlier quoted context omitted.

New filesystems seems to have a chicken and egg problem really. It's not like switching from Nvidia's proprietary drivers to nouveau and then back if it turns out they don't work that well. Switching filesystems, especially in larger raid setups where you desperately need more testing and real world usage feedback, is pretty involved, and even if you have everything backed up it's pretty time consuming restoring ever…

> In an ideal world, you'd nail your FS design first try, make no mistakes during implementation and call it a day Crypto implementations and FS implementations strike me as the ideal audience for actually investing the mental energy in the healthy ecosystem we have of modeling and correctness verification systems Now, I readily admit that I could be talking out of my ass, given that I've not tried to use those verif…

A major chunk of storage reliability is all these weird and unexpected failure modes and edge cases which are not possible to prepare for, let alone write fixed specs for. Software correctness assumes the underlying system behaves correctly and stays fixed, which is not the case. You can't trust the hardware and the systems are too diverse - this is the worst case for formal verification.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#288

Post by louwrentius »

mkeeter wrote: post #245
louwrentius wrote: post #51

Earlier quoted context omitted.

+1 for mentioning 0xide. I love that they went this route and that stat is interesting. I hate the typical DC high RPM small fan whine. I also hope that they do something 'smart' when they control the fan speed ;-)

It’s moderately smart - there’s a PID loop with per-component target temperatures, so it’s trying not to do more work than necessary. (source: I wrote it, and it’s all published at https://github.com/oxidecomputer/hubris/tree/master/task/the... ) We also worked with the fan vendor to get parts with a lower minimum RPM. The stock fans idle at about 5K RPM, and ours idle at 2K, which is already enough to keep the syste…

Ha! thanks a lot for sharing, love it. Nice touch to use low idle RPM fans.

Same thing for my ancient NAS: after boot, the fans run at idle for hours and the PID controller just doesn't have to do anything at all.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#289

Post by kenhwang »

Kirby64 wrote: post #116
kenhwang wrote: post #82

Earlier quoted context omitted.

They're airtight now (at the high end or enterprise level). They weren't airtight not very long ago and had filters to regulate the air exchange.

They're not airtight in the true sense (besides the helium filled ones nowadays), but every drive made in the past... 30? 40 years is airtight in the sense that no dust can ever get into the drive. There's a breather hole somewhere (with a big warning to not cover it!) to equalize pressure, and a filter that doesn't allow essentially any particles in.

No dust is supposed to get "in" the drive, but dust can very well clog the breather hole and cause pressure issues that could kill the drive.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#290

Post by Kirby64 »

kenhwang wrote: post #289
Kirby64 wrote: post #116

Earlier quoted context omitted.

They're not airtight in the true sense (besides the helium filled ones nowadays), but every drive made in the past... 30? 40 years is airtight in the sense that no dust can ever get into the drive. There's a breather hole somewhere (with a big warning to not cover it!) to equalize pressure, and a filter that doesn't allow essentially any particles in.

No dust is supposed to get "in" the drive, but dust can very well clog the breather hole and cause pressure issues that could kill the drive.

Unless you’re moving the altitude of the drive substantially after it’s already clogged, how would this happen? There’s no air exchange on hard drives.
Post reply on HN