Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: ZFS silent data corruption bugfix – my research results

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#81

Post by sgarland »

3np wrote: post #34
buildbot wrote: post #10

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893 , which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun. Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable. Let's do away with the tribalism in both ways? Calling…

As someone currently rewriting Ansible plays to build Bookworm QEMU images with ZFS support, thank you for this incredibly niche and useful bit of information.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#82

Post by buildbot »

worthless-trash wrote: post #79
lproven wrote: post #46

Earlier quoted context omitted.

ext4 released: 2008 https://archive.ph/20120529150649/http://git.kernel.org/?p=l... ZFS released: 2005 https://web.archive.org/web/20130619165135/https://blogs.ora... This is Linux local bias. There is more to Unix than Linux. Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on al…

I would -very- strongly imagine it the other way. I guess we'll never know.

I worked at a company that shipped petabytes of ZFS storage for years and years. Think 42u racks with 60 drive JBODs. Hundreds of these. Hundreds. And they were a very small company. Can you imagine the scale Sun and Oracle shipped hardware at? Was there any linux based hardware company operating at the same scale?

I don’t think we had any customers request ext4 bulk data storage. OS drives sure. Everything else was ZFS. Or weirder proprietary filesystems!

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#83

Post by 3np »

sgarland wrote: post #81
3np wrote: post #34

Earlier quoted context omitted.

My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893 , which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun. Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable. Let's do away with the tribalism in both ways? Calling…

As someone currently rewriting Ansible plays to build Bookworm QEMU images with ZFS support, thank you for this incredibly niche and useful bit of information.

Np! Actually, in case you're able to share some of those plays in any form, that could be very useful here, however niche :)

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#84

Post by lproven »

worthless-trash wrote: post #79
lproven wrote: post #46

Earlier quoted context omitted.

ext4 released: 2008 https://archive.ph/20120529150649/http://git.kernel.org/?p=l... ZFS released: 2005 https://web.archive.org/web/20130619165135/https://blogs.ora... This is Linux local bias. There is more to Unix than Linux. Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on al…

I would -very- strongly imagine it the other way. I guess we'll never know.

See the earlier comment to you.

I think you have this 100% backwards. I heard similar comments to yours inside some large Linux vendors, too, and I think they betray a profound misunderstanding about the nature of large-scale Unix deployments over the years.

FOSS Unix is recent and mostly small-scale and transient deployments of containers, not long-term storage... but before that, there were decades of vast scale commercial Unix systems, which dwarf the FOSS stuff.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#85

Post by sgarland »

3np wrote: post #83
sgarland wrote: post #81

Earlier quoted context omitted.

As someone currently rewriting Ansible plays to build Bookworm QEMU images with ZFS support, thank you for this incredibly niche and useful bit of information.

Np! Actually, in case you're able to share some of those plays in any form, that could be very useful here, however niche :)

I have an older version [0] for Bullseye that you're welcome to take a look at. It's actually combined Ansible + Packer, so if you point it at a Proxmox host, you get an image in ~10-15 minutes.

It's a fork of other work [1], modified and updated for Debian and Debian only. It also in absolutely no way has any best practices, which is why I'm doing a complete re-write with proper role dependencies, modularity, etc. When that's done, I'll be marking [0] as archived and linking to the new repo.

The new version also replaces the incredibly hacky shell script builder (hooray dialog) with Python, and updates Packer's JSON with HCL2.

[0]: https://github.com/stephanGarland/packer-proxmox-templates

[1]: https://github.com/chriswayg/packer-proxmox-templates

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#86

Post by 3np »

sgarland wrote: post #85
3np wrote: post #83

Earlier quoted context omitted.

Np! Actually, in case you're able to share some of those plays in any form, that could be very useful here, however niche :)

I have an older version [0] for Bullseye that you're welcome to take a look at. It's actually combined Ansible + Packer, so if you point it at a Proxmox host, you get an image in ~10-15 minutes. It's a fork of other work [1], modified and updated for Debian and Debian only. It also in absolutely no way has any best practices, which is why I'm doing a complete re-write with proper role dependencies, modularity, etc. W…

Thank you!

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#87

Post by worthless-trash »

buildbot wrote: post #82
worthless-trash wrote: post #79

Earlier quoted context omitted.

I would -very- strongly imagine it the other way. I guess we'll never know.

I worked at a company that shipped petabytes of ZFS storage for years and years. Think 42u racks with 60 drive JBODs. Hundreds of these. Hundreds. And they were a very small company. Can you imagine the scale Sun and Oracle shipped hardware at? Was there any linux based hardware company operating at the same scale? I don’t think we had any customers request ext4 bulk data storage. OS drives sure. Everything else was…

Yeah, that GPFS was real pita, i'm not talking about size comparison issues. I'm talking about runtime in the field.

You simply can't begin to understand the amount of shitty iot and android devices that exist.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#88

Post by worthless-trash »

lproven wrote: post #84
worthless-trash wrote: post #79

Earlier quoted context omitted.

I would -very- strongly imagine it the other way. I guess we'll never know.

See the earlier comment to you. I think you have this 100% backwards. I heard similar comments to yours inside some large Linux vendors, too, and I think they betray a profound misunderstanding about the nature of large-scale Unix deployments over the years. FOSS Unix is recent and mostly small-scale and transient deployments of containers, not long-term storage... but before that, there were decades of vast scale co…

How many zfs systems you think are running right now, all hours combined compared to linux ext filesystems ?

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#89

Post by lproven »

worthless-trash wrote: post #88
lproven wrote: post #84

Earlier quoted context omitted.

See the earlier comment to you. I think you have this 100% backwards. I heard similar comments to yours inside some large Linux vendors, too, and I think they betray a profound misunderstanding about the nature of large-scale Unix deployments over the years. FOSS Unix is recent and mostly small-scale and transient deployments of containers, not long-term storage... but before that, there were decades of vast scale co…

How many zfs systems you think are running right now, all hours combined compared to linux ext filesystems ?

"Bananas are better than passion fruit because there are more bananas being grown!"

ZFS is an enterprise storage-management tool. It's not just another hard disk format, like ext4.

Not many people are running Linux boxes with root on ZFS because few distros support that.

Single-disk setups are not its strength so the number of single-disk setups of ZFS compared to ext4, which only supports single-disk setups, is irrelevant.

Even saying that, though...

I bet all FreeBSD deployments are on ZFS, and it's got a solid enterprise presence. Anything streamed off Netflix anywhere is being served by FreeBSD.

Oracle still supports Solaris. All Solaris, OpenSolaris, Illumos, Nexenta, OmniOPS, SmartOS, Tribblix... all on ZFS.

And it's still not its core area.

If you want to do a fair comparison let's hear about companies doing petascale to exascale arrays with double-digit to quadruple-digit numbers of drives.

How many of them do you reckon use ext4?

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#90

Post by worthless-trash »

lproven wrote: post #89
worthless-trash wrote: post #88

Earlier quoted context omitted.

How many zfs systems you think are running right now, all hours combined compared to linux ext filesystems ?

"Bananas are better than passion fruit because there are more bananas being grown!" ZFS is an enterprise storage-management tool. It's not just another hard disk format, like ext4. Not many people are running Linux boxes with root on ZFS because few distros support that. Single-disk setups are not its strength so the number of single-disk setups of ZFS compared to ext4, which only supports single-disk setups, is irre…

I'm not the one who moved the goal posts here though was I ?

Here, let me quote myself.

> The software is being used by more people, which means it will expose more bugs.

Post reply on HN