Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

832 TB – ZFS on Linux

jonkensy.com

Post reply on HN
Print view View original

121–130 of 165 posts

Re: 832 TB – ZFS on Linux

#121

Post by jjirsa »

fnj wrote: post #95
jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

> Zfs on Linux has data loss bugs. Please don't spread untruths. Somebody who doesn't know better might actually believe you.

https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/160...

Unpatched on 16.04, referenced as supported in the article

Re: 832 TB – ZFS on Linux

#122

Post by jjirsa »

codys wrote: post #72
jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

> There's at least one unpatched and there are bound to be more. References, please.

https://news.ycombinator.com/item?id=14938092

Re: 832 TB – ZFS on Linux

#123

Post by jjirsa »

codys wrote: post #73
jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

It might be worthwhile mentioning that at some point in the past, HN ran on 1 server (and might still do so). https://news.ycombinator.com/item?id=5229522

Cute but irrelevant?

Re: 832 TB – ZFS on Linux

#124

Post by olavgg »

matt_wulfeck wrote: post #116
kev009 wrote: post #93

There are a couple needful tweaks to this BOM for anyone wanting to follow this.. Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality. I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or…

Why FreeBSD?

The reason is that the Linux and FreeBSD kernels are vastly different. The FreeBSD kernel is more similar to Solaris than Linux and this is what ZFS was developed for.

So in the end you have way more kernel "workarounds" for ZFS on Linux than the FreeBSD implementation. Especially the virtual memory implementation on Linux behaves very differently and has caused many issues over time, but I think it is very good now.

Re: 832 TB – ZFS on Linux

#126

Post by drvdevd »

exikyut wrote: post #85
jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

I'm planning on doing a tiny ZFS pool (once I've finally saved up for it - financials can be fun sometimes!), and was thinking of doing FreeBSD for ZFS and Linux for everything else on top of Xen. I'm currently unsure how to make Linux see the ZFS pool though. I.... don't really like NFS. It's too glitchy in my experience. I use it to listen to music stored on a different machine from my laptop, which uses a long-ran…

You could in theory have a zvol exported from a ZFS pool on FreeBSD to Linux via some remote block dev protocol. iSCSI comes to mind, as it would have to be cross platform. Then slap ext4 on top of that in Linux.

The idea feels "janky" though. Lots of overhead compared to just running ZFS on Linux.

Re: 832 TB – ZFS on Linux

#127

Post by atmosx »

tedunangst wrote: post #92
atmosx wrote: post #80

Earlier quoted context omitted.

> For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux That's the Nth I've read this quote on HN, it became a classic... You can't find a FreeBSD sysadmin but you can find a Linux admin. Where I work I have to deal with AIX, Solaris, Open/FreeBSD (Had Net before), Linux (all major flavours) and (god Forbid) Windows Server (2008, 12R2, 2016 and Nano). I've build packages for mos…

Guess it depends on the flavor of admin. With some regularity a linux admin "with decades of experience" will show up and announce that openbsd is terribly broken and nothing works, not even the most basic pkg-add command. Uh, did you mean pkg_add? See! Openbsd is so broken they called the command pkg_add while I typed pkg-add. I've never had this trouble with linux! I'd be worried about letting such a person admin l…

Totally agree. BTW, thanks for signify it is an awesome system for pkg signatures. It took me a while to get it working correctly for automatic pkg signing but once it all clicked together, the system was simple as any I've see!

ps. The best documentation I've found, was the manpage[1].

[1] https://man.openbsd.org/signify

Re: 832 TB – ZFS on Linux

#128

Post by cyphar »

jjirsa wrote: post #121
fnj wrote: post #95

Earlier quoted context omitted.

> Zfs on Linux has data loss bugs. Please don't spread untruths. Somebody who doesn't know better might actually believe you.

https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/160... Unpatched on 16.04, referenced as supported in the article

I read the patch and linked issues. It looks like an OpenZFS bug to me, which wasn't ported to ZoL until recently but it definitely doesn't look like a ZoL-exclusive bug.

Not to mention that the issue description doesn't say it's a data loss bug. It's a bug that means that ZFS send would not include holes in very specific scenarios. On-disk data would still be safe as far as I can see (though I'm not an FS expert by any stretch).

Re: 832 TB – ZFS on Linux

#129

Post by cyphar »

FullyFunctional wrote: post #117
deelowe wrote: post #112

Earlier quoted context omitted.

Been running an 8 drive array for nearly 5 years now myself and never once had an issue through 3 PCs and 3 different operating systems. What was your issue?

With utmost respect: this isn't a super valuable data point. A "production deployment" with lots of users and/or workloads will see issues you will never encounter in a moderate setting. FWIW, I use FreeNAS in a similar small but diverse setting with nary a problem, but when I sat it up for a 30+ organization, issue arose that I hadn't expected (not data loss, but usability and performance issues).

ZFS has definitely been used in very large production deployments in a variety of scenarios for well over a decade. The question seems reasonable given that history. ZFS isn't some toy filesystem hacked together over a weekend.

Re: 832 TB – ZFS on Linux

#130

Post by XorNot »

exikyut wrote: post #90
patrickg_zill wrote: post #79

Earlier quoted context omitted.

Actually the real issue is, "when the system is 65% full and you reboot, how long will it take for ZFS to mount it"? Perhaps he has split up ZFS into a number of different pools and they can be mounted in parallel (depends on the init script and whether ZFS can do this). But I do recall that larger ZFS pools can take a bit of time to mount; maybe the updated ZFS for Linux is faster....

> how long will it take for ZFS to mount it? What? How long would it take, roughly? Genuinely curious.

Depends on the number of snapshots in my experience. 84,000 was slightly too many.
Post reply on HN