Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Working with Files Is Hard (2019)

danluu.com

Post reply on HN
Print view View original

111–120 of 123 posts

Re: Working with Files Is Hard (2019)

#111

Post by [deleted] »

ein0p wrote: post #46
ryao wrote: post #31

> On Linux ZFS, it appears that there's a code path designed to do the right thing, but CPU usage spikes and the system may hang or become unusable. ZFS fsync will not fail, although it could end up waiting forever when a pool faults due to hardware failures: https://papers.freebsd.org/2024/asiabsdcon/norris_openzfs-fs...

ZFS on Linux unfortunately has a long standing bug which makes it unusable under load: https://github.com/openzfs/zfs/issues/9130 . 5.5 years old, nobody knows the root cause. Symptoms: under load (such as what one or two large concurrent rsyncs may generate over a fast network - that's how I encountered it) the pool begins to crap out and shows integrity errors and in some cases loses data (for some users - it never…

[deleted]

Re: Working with Files Is Hard (2019)

#112

Post by ryao »

ein0p wrote: post #46
ryao wrote: post #31

> On Linux ZFS, it appears that there's a code path designed to do the right thing, but CPU usage spikes and the system may hang or become unusable. ZFS fsync will not fail, although it could end up waiting forever when a pool faults due to hardware failures: https://papers.freebsd.org/2024/asiabsdcon/norris_openzfs-fs...

ZFS on Linux unfortunately has a long standing bug which makes it unusable under load: https://github.com/openzfs/zfs/issues/9130 . 5.5 years old, nobody knows the root cause. Symptoms: under load (such as what one or two large concurrent rsyncs may generate over a fast network - that's how I encountered it) the pool begins to crap out and shows integrity errors and in some cases loses data (for some users - it never…

There are actually several distinct issues being reported there. I replied responding to everyone who posted backtraces and a few who did not:

https://github.com/openzfs/zfs/issues/9130#issuecomment-2614...

That said, there are many others who stress ZFS on a regular basis and ZFS handles the stress fine. I do not doubt that there are bugs in the code, but I feel like there are other things at play in that report. Messages saying that the txg_sync thread has hung for 120 seconds typically indicate that disk IO is running slowly due to reasons external to ZFS (and sometimes, reasons internal to ZFS, such as data deduplication).

I will try to help everyone in that issue. Thanks for bringing that to my attention. I have been less active over the past few years, so I was not aware of that mega issue.

Re: Working with Files Is Hard (2019)

#113

Post by ryao »

huntaub wrote: post #60
ryao wrote: post #39

Earlier quoted context omitted.

Packages consist of multiple files. An atomic file write would not allow packages to be either installed or not installed by APT.

This is kind of an interesting thought that more mirrors how Docker uses OverlayFS to track changes to the entire file system. No need for new file APIs.

It can also use ZFS to do this.

Re: Working with Files Is Hard (2019)

#114

Post by ein0p »

ryao wrote: post #112
ein0p wrote: post #46

Earlier quoted context omitted.

ZFS on Linux unfortunately has a long standing bug which makes it unusable under load: https://github.com/openzfs/zfs/issues/9130 . 5.5 years old, nobody knows the root cause. Symptoms: under load (such as what one or two large concurrent rsyncs may generate over a fast network - that's how I encountered it) the pool begins to crap out and shows integrity errors and in some cases loses data (for some users - it never…

There are actually several distinct issues being reported there. I replied responding to everyone who posted backtraces and a few who did not: https://github.com/openzfs/zfs/issues/9130#issuecomment-2614... That said, there are many others who stress ZFS on a regular basis and ZFS handles the stress fine. I do not doubt that there are bugs in the code, but I feel like there are other things at play in that report. Me…

Regarding your comment - seems unlikely that it "affects Ubuntu less". I don't see why that would be the case - it's not like Ubuntu runs a heavily customized kernel or anything. And thanks for taking a look - ZFS is just the way things should be in filesystems and logical volume management, I do wish I could stop doing hash compares after large, high throughput copies and just trust it to do what it was designed to do.

Re: Working with Files Is Hard (2019)

#115

Post by ryao »

ein0p wrote: post #114
ryao wrote: post #112

Earlier quoted context omitted.

There are actually several distinct issues being reported there. I replied responding to everyone who posted backtraces and a few who did not: https://github.com/openzfs/zfs/issues/9130#issuecomment-2614... That said, there are many others who stress ZFS on a regular basis and ZFS handles the stress fine. I do not doubt that there are bugs in the code, but I feel like there are other things at play in that report. Me…

Regarding your comment - seems unlikely that it "affects Ubuntu less". I don't see why that would be the case - it's not like Ubuntu runs a heavily customized kernel or anything. And thanks for taking a look - ZFS is just the way things should be in filesystems and logical volume management, I do wish I could stop doing hash compares after large, high throughput copies and just trust it to do what it was designed to…

Ubuntu kernels might have a different default IO elevator than proxmox kernels. If the issue is in the IO elevator (e.g. it is reordering in such a way that some IOs are delayed indefinitely before being sent to the underlying device) and the two use different IO elevators by default, then it would make sense why Ubuntu is not affected and proxmox is. There is some evidence for this in the comments as people suggest that the issue is lessened by switching to mq-deadline. That is why one of my questions asks what Linux IO elevator people’s disks are using.

The correct IO elevator to use for disks given to ZFS is none/noop as ZFS has its own IO elevator. ZFS will set the Linux IO elevator to that automatically on disks where it controls the partitioning. However, when the partitioning was done externally from ZFS, the default Linux elevator is used underneath ZFS, and that is never none/noop in practice since other Linux filesystems benefit from other elevators. If proxmox is doing partitioning itself, then it is almost certainly using the wrong IO elevator with ZFS, unless it sets the elevator to noop when ZFS is using the device. That ordinarily should not cause such severe problems, but it is within the realm of possibility that the Linux IO elevator being set by proxmox has a bug.

I suspect there are multiple disparate issues causing the txg_sync thread to hang for people, rather than just one issue. Historically, things that cause the txg_sync thread to hang are external to ZFS (with the notable exception of data deduplication), so it is quite likely that the issues are external here too. I will watch the thread and see what feedback I get from people who are having the txg_sync thread hang.

Re: Working with Files Is Hard (2019)

#116

Post by ein0p »

ryao wrote: post #115
ein0p wrote: post #114

Earlier quoted context omitted.

Regarding your comment - seems unlikely that it "affects Ubuntu less". I don't see why that would be the case - it's not like Ubuntu runs a heavily customized kernel or anything. And thanks for taking a look - ZFS is just the way things should be in filesystems and logical volume management, I do wish I could stop doing hash compares after large, high throughput copies and just trust it to do what it was designed to…

Ubuntu kernels might have a different default IO elevator than proxmox kernels. If the issue is in the IO elevator (e.g. it is reordering in such a way that some IOs are delayed indefinitely before being sent to the underlying device) and the two use different IO elevators by default, then it would make sense why Ubuntu is not affected and proxmox is. There is some evidence for this in the comments as people suggest…

Thanks a lot for elaborating. I'm traveling at the moment, but I'm going to try reproducing this issue once I'm back in town. IIRC I did do partitioning myself, using GPT partition table and default partition settings in fdisk.

Upd mq-deadline for all drives seems to be `none` for me. OS is Ubuntu 22.04

Re: Working with Files Is Hard (2019)

#117

Post by ryao »

ein0p wrote: post #116
ryao wrote: post #115

Earlier quoted context omitted.

Ubuntu kernels might have a different default IO elevator than proxmox kernels. If the issue is in the IO elevator (e.g. it is reordering in such a way that some IOs are delayed indefinitely before being sent to the underlying device) and the two use different IO elevators by default, then it would make sense why Ubuntu is not affected and proxmox is. There is some evidence for this in the comments as people suggest…

Thanks a lot for elaborating. I'm traveling at the moment, but I'm going to try reproducing this issue once I'm back in town. IIRC I did do partitioning myself, using GPT partition table and default partition settings in fdisk. Upd mq-deadline for all drives seems to be `none` for me. OS is Ubuntu 22.04

> Upd mq-deadline for all drives seems to be `none` for me.

I am not sure what you mean by that. One possibility is that the ones who reported mq-deadline did better were on either kyber or bfq, rather than none. The none elevator should be best for ZFS.

Re: Working with Files Is Hard (2019)

#118

Post by ein0p »

ryao wrote: post #117
ein0p wrote: post #116

Earlier quoted context omitted.

Thanks a lot for elaborating. I'm traveling at the moment, but I'm going to try reproducing this issue once I'm back in town. IIRC I did do partitioning myself, using GPT partition table and default partition settings in fdisk. Upd mq-deadline for all drives seems to be `none` for me. OS is Ubuntu 22.04

> Upd mq-deadline for all drives seems to be `none` for me. I am not sure what you mean by that. One possibility is that the ones who reported mq-deadline did better were on either kyber or bfq, rather than none. The none elevator should be best for ZFS.

I mean that it's already "none" on the machine where I encountered this bug. "Upd" was merely to signal that I've made an edit to my post.

Re: Working with Files Is Hard (2019)

#119

Post by ryao »

ein0p wrote: post #118
ryao wrote: post #117

Earlier quoted context omitted.

> Upd mq-deadline for all drives seems to be `none` for me. I am not sure what you mean by that. One possibility is that the ones who reported mq-deadline did better were on either kyber or bfq, rather than none. The none elevator should be best for ZFS.

I mean that it's already "none" on the machine where I encountered this bug. "Upd" was merely to signal that I've made an edit to my post.

The mq-deadline part is what confused me. That is a competing option for the Linux IO elevator that runs under ZFS. Anyway, I understand now and thanks for the data point. I added a list of questions to GitHub that could help narrow things down if you take time to answer them. I will be watching the GitHub thread regularly, so I will see it if you post the answers after you return from your travels.

Re: Working with Files Is Hard (2019)

#120

Post by ryao »

timewizard wrote: post #41
josephg wrote: post #32

Earlier quoted context omitted.

It doesn’t have to be one or the other. Developers could decide by passing flags to open. But even then, doing atomic writes of multi gigabyte files doesn’t sound that hard to implement efficiently. Just write to disk first and update the metadata atomically at the end. Or whenever you choose to as a programmer. The downside is that, when overwriting, you’ll need enough free space to store both the old and new versio…

> Developers could decide by passing flags to open. Provided the underlying VFS has implemented them. They may not. Hence the point in the article that some developers only choose to support 'ext4' and nothing else. > you’ll need enough free space to store both the old and new versions of your data. The sacrifice is increased write wear on solid state devices. > It would allow all sorts of useful programs to be writt…

Dropbox reversed its stance on this. It added support for ZFS, XFS, ecryptfs and btrfs:

https://help.dropbox.com/installs/system-requirements

They say ecryptfs is only supported when it is backed by ext4, which is a bit strange. I wonder if that is documented just to be able to close support cases when ecryptfs is used on top of a filesystem that is missing extended attribute support and their actual code does not actually check what is below ecryptfs. Usually the application above would not know what is below ecryptfs, so they would need to go out of their way to check this in order to enforce that. I do not use Dropbox, so someone else would need to test to see if they actually enforce that if curious enough.

Post reply on HN