Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

832 TB – ZFS on Linux

jonkensy.com

Post reply on HN
Print view View original

91–100 of 165 posts

Re: 832 TB – ZFS on Linux

#91

Post by ams6110 »

jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

My (admittedly limted) experience with ZFS has been bad. I know it's supposed to be great but that is not what I have experienced. Multi-hour outages. Having to have Oracle help resolve problems. This was a few year ago now, maybe it's better these days. I was not the administrator, just a user, but we had ZFS issues at least quarterly on a ~20TB shared filesystem on a Linux cluster.

Re: 832 TB – ZFS on Linux

#92

Post by tedunangst »

atmosx wrote: post #80
notpeter wrote: post #21

Earlier quoted context omitted.

It's likely a human resources problem. For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux. And those numbers are much worse outside of major cities. The commercial support from Ubuntu tips the scales. I made this same decision at my last job. I ran Solaris and Illumos on our file servers and loved it, but a year before I left I ported all the pools to Ubuntu so my successor o…

> For every competent FreeBSD or Illumos sysadmin there are 10x equally experienced with Linux That's the Nth I've read this quote on HN, it became a classic... You can't find a FreeBSD sysadmin but you can find a Linux admin. Where I work I have to deal with AIX, Solaris, Open/FreeBSD (Had Net before), Linux (all major flavours) and (god Forbid) Windows Server (2008, 12R2, 2016 and Nano). I've build packages for mos…

Guess it depends on the flavor of admin. With some regularity a linux admin "with decades of experience" will show up and announce that openbsd is terribly broken and nothing works, not even the most basic pkg-add command. Uh, did you mean pkg_add? See! Openbsd is so broken they called the command pkg_add while I typed pkg-add. I've never had this trouble with linux!

I'd be worried about letting such a person admin linux servers, but I guess you can limp by as long as you keep your infra to what they already know. Ideally you'd weed such people out before hiring, but maybe if you need to hire an admin you don't know enough to do that?

Re: 832 TB – ZFS on Linux

#93

Post by kev009 »

There are a couple needful tweaks to this BOM for anyone wanting to follow this..

Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality.

I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or zero copy TCP.

Supermicro seriously under-provisioned I/O on that chassis. I'd add LSI/Avago/now Broadcom cards so you can get native ports to every drive. Even if it's just a cold storage box, it will help with rebuild and scrub times and peace of mind. The cost of this is not bad compared to the frustration of SAS expander firmwares. 2x24 or 3x16 and 4 drives on the onboard if you can skip the backplane expander. Supermicro will usually do things like this if you insist, or an integrator like ixSystems can handle it.

More subjectively, I would also recommend FreeBSD. It seems their main justification for Ubuntu was paid support, which can be had from ixSystems who sell and support an entire stack (Supermicro servers, FreeBSD or FreeNAS or TrueNAS, and grok ZFS and storage drivers to the tune that they have done quite a bit of development.

Re: 832 TB – ZFS on Linux

#94

Post by cyphar »

usefulcat wrote: post #81
tylerjl wrote: post #24

Earlier quoted context omitted.

I'm only a casual user of ZFS on Linux for personal storage projects, but I've spoken with people who rely very heavily on ZFS on Linux for their small businesses, and it's interesting to hear their perspectives on this. Essentially, because btrfs has failed to deliver on the next-gen filesystem front, ZFS on Linux is such a critical piece of technology that unless Red Hat has an answer soon for out-of-the-box ZFS on…

RHEL is not continuing to bet on btrfs; they are deprecating it: https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...

As was explained in depth in the HN comments, Red Hat's decision to not provide enterprise support is based on the fact that they don't have the necessary engineers to backport and maintain one of the fastest moving pieces of code in Linux today. All of their btrfs engineers moved to Facebook, and they're still working on btrfs improvements.

There are many decisions involved in deciding what to provide enterprise support for, and whether you like it or not, technical merit is only one of many factors. So Red Hat's decision was likely not entirely based on technical merit.

SUSE still provides enterprise support for btrfs (like we've always done), and there's still plenty of work being done from various large contributors.

[I work at SUSE.]

Re: 832 TB – ZFS on Linux

#95

Post by fnj »

jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

> Zfs on Linux has data loss bugs.

Please don't spread untruths. Somebody who doesn't know better might actually believe you.

Re: 832 TB – ZFS on Linux

#96

Post by [deleted] »

rsync wrote: post #7

"I ended up between the Supermicro SSG-6048R-E1CR60L or the SSG-6048R-E1CR90L – the E1CR60L is a 60-bay 4U chassis while the E1CR90L is a 90-bay 4U chassis. This nice part is that no matter which platform you choose Supermicro sells this only as a pre-configured machine – this means that their engineers are going to make sure that the hardware you choose to put in this is all from a known compatibility list. Basicall…

[deleted]

Re: 832 TB – ZFS on Linux

#97

Post by [deleted] »

rsync wrote: post #7

"I ended up between the Supermicro SSG-6048R-E1CR60L or the SSG-6048R-E1CR90L – the E1CR60L is a 60-bay 4U chassis while the E1CR90L is a 90-bay 4U chassis. This nice part is that no matter which platform you choose Supermicro sells this only as a pre-configured machine – this means that their engineers are going to make sure that the hardware you choose to put in this is all from a known compatibility list. Basicall…

[deleted]

Re: 832 TB – ZFS on Linux

#98

Post by waz0wski »

cyphar wrote: post #17
qeternity wrote: post #16

Earlier quoted context omitted.

Sure but that alone doesn't mean it's any less suitable on Ubuntu. I'm just trying to understand what the differences are in implementation, reliability, etc.

ZFS-on-Linux has not been mainstream (or existed) for as long as ZFS has been integrated inside FreeBSD. illumos is the free software fork of OpenSolaris -- so it's the repo of record for modern ZFS development.

OpenZFS is the parent project from which ZFS is applied to Illumos, FreeBSD, and ZFSoL

http://open-zfs.org/wiki/Main_Page

Re: 832 TB – ZFS on Linux

#99

Post by cyphar »

waz0wski wrote: post #98
cyphar wrote: post #17

Earlier quoted context omitted.

ZFS-on-Linux has not been mainstream (or existed) for as long as ZFS has been integrated inside FreeBSD. illumos is the free software fork of OpenSolaris -- so it's the repo of record for modern ZFS development.

OpenZFS is the parent project from which ZFS is applied to Illumos, FreeBSD, and ZFSoL http://open-zfs.org/wiki/Main_Page

I'm aware of that, I just didn't want to additionally confuse someone who is barely familiar with ZFS with the whole OpenZFS fork and so on.
Post reply on HN