Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

832 TB – ZFS on Linux

jonkensy.com

Post reply on HN
Print view View original

111–120 of 165 posts

Re: 832 TB – ZFS on Linux

#111

Post by deelowe »

jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

You care to back up those data loss claims with any proof?

I agree that this smells a bit funny, but the author seems to dance around the important topics like vpools, raid levels, and use case. Instead they just focus on the hardware, which is arguably the least interesting bit (until you know the former).

Re: 832 TB – ZFS on Linux

#112

Post by deelowe »

ams6110 wrote: post #91
jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

My (admittedly limted) experience with ZFS has been bad. I know it's supposed to be great but that is not what I have experienced. Multi-hour outages. Having to have Oracle help resolve problems. This was a few year ago now, maybe it's better these days. I was not the administrator, just a user, but we had ZFS issues at least quarterly on a ~20TB shared filesystem on a Linux cluster.

Been running an 8 drive array for nearly 5 years now myself and never once had an issue through 3 PCs and 3 different operating systems. What was your issue?

Re: 832 TB – ZFS on Linux

#113

Post by deelowe »

comboy wrote: post #29
jaytaylor wrote: post #27

What about cooling? Will the lifespan of the high-capacity platter-dense hard drives be drastically reduced by clumping them together like that with what looks like little airflow?

AFAIR from the backblaze blog, a bigger issue that shortens the lifespan is vibration.

That's probably because their design doesn't do much to mitigate it.

Re: 832 TB – ZFS on Linux

#114

Post by petre »

cmurf wrote: post #45
tylerjl wrote: post #24

Earlier quoted context omitted.

I'm only a casual user of ZFS on Linux for personal storage projects, but I've spoken with people who rely very heavily on ZFS on Linux for their small businesses, and it's interesting to hear their perspectives on this. Essentially, because btrfs has failed to deliver on the next-gen filesystem front, ZFS on Linux is such a critical piece of technology that unless Red Hat has an answer soon for out-of-the-box ZFS on…

SUSE has succeeded in shipping atomic OS updates and rollbacks by default both for enterprise and community versions of Linux several years ago. Facebook is using it, and the usage is growing. [1] And there are any number of examples of production usage of Btrfs at scale if you go look for them. [2] And the good and bad are fairly well understood, it is in fact getting better and will continue to get better.[3] And R…

Aren't Synology NAS devices also using Brtfs?

Re: 832 TB – ZFS on Linux

#115

Post by petre »

jjirsa wrote: post #61

Zfs on linux and huge single servers, what could go wrong? It's like a blog written by a 22 year old straight out of college that's never dealt with a real production deployment/failure Zfs on Linux has data loss bugs. There's at least one unpatched and there are bound to be more. Single huge servers eventually fail. Maybe it'll be a drive controller. Maybe it'll be CPU or ram with bit flips as a side effect. Downtim…

> It's like a blog written by a 22 year old straight out of college

Also no mention of FreeBSD. Instead he picked the less battle tested ZoL and only mentioned Solaris and licensing, so the author is not even aware of Illumos.

Re: 832 TB – ZFS on Linux

#116

Post by matt_wulfeck »

kev009 wrote: post #93

There are a couple needful tweaks to this BOM for anyone wanting to follow this.. Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality. I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or…

Why FreeBSD?

Re: 832 TB – ZFS on Linux

#117

Post by FullyFunctional »

deelowe wrote: post #112
ams6110 wrote: post #91

Earlier quoted context omitted.

My (admittedly limted) experience with ZFS has been bad. I know it's supposed to be great but that is not what I have experienced. Multi-hour outages. Having to have Oracle help resolve problems. This was a few year ago now, maybe it's better these days. I was not the administrator, just a user, but we had ZFS issues at least quarterly on a ~20TB shared filesystem on a Linux cluster.

Been running an 8 drive array for nearly 5 years now myself and never once had an issue through 3 PCs and 3 different operating systems. What was your issue?

With utmost respect: this isn't a super valuable data point. A "production deployment" with lots of users and/or workloads will see issues you will never encounter in a moderate setting.

FWIW, I use FreeNAS in a similar small but diverse setting with nary a problem, but when I sat it up for a 30+ organization, issue arose that I hadn't expected (not data loss, but usability and performance issues).

Re: 832 TB – ZFS on Linux

#118

Post by b34r »

matt_wulfeck wrote: post #116
kev009 wrote: post #93

There are a couple needful tweaks to this BOM for anyone wanting to follow this.. Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality. I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or…

Why FreeBSD?

stability.

Re: 832 TB – ZFS on Linux

#120

Post by electrum »

cmurf wrote: post #33

Do either of these project spec hardware that would work for this use case? opencompute.org Backblaze storage pod, they're up to v 6.0 now (Netflix open connect specs supermicro hardware) Others?

The next generation Facebook design for storage is Bryce Canyon, available through the Open Compute Project: https://code.facebook.com/posts/1869788206569924/introducing...
Post reply on HN