Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

151–160 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#151

Post by db48x »

rincebrain wrote: post #59
rincebrain wrote: post #56

Earlier quoted context omitted.

First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris. _Some_ of the Linux systems had a problem where…

A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I qu…

Disk drives really are the worst. Someone who works at Google or Amazon or The Internet Archive could (and should!) write a book where every single chapter tells the story of discovering some new batch of hard drives with a new and crazy firmware bug.

Re: In OpenZFS and Btrfs, everyone was just guessing

#152

Post by mischief6 »

GrumpySloth wrote: post #144
RantyDave wrote: post #37

Earlier quoted context omitted.

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…

> disabling the writing of 'last accessed' timestamps atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime.

while it is unfortunate it requires a write, it is useful in some cases. for example, finding old objects in a cache, or today i used it to find out what firmware my kernel is loading by checking atime of /lib/firmware.

Re: In OpenZFS and Btrfs, everyone was just guessing

#153

Post by Too »

admax88qqq wrote: post #146
jcalvinowens wrote: post #135

Earlier quoted context omitted.

> So user's are at fault for trusting a filesystem that advertises itself as being reliable? Yes, absolutely 100% at fault. There are many failure modes here beyond corruption: human error, electrical surges, flooding, fire. Hell, somebody could break into your house and steal the NAS! Filesystem bugs aren't a big deal. If they are a big deal to you, it's because you're trusting a system with a single point of failur…

Could you imagine if we treated other pieces of technology the same way backup fanatics talk about backups? It's your fault for not having 2 extra phones on you in case your phone bricks itself at a bad time. It's your fault for not encrypting your signal messages by hand, encryption bugs aren't a big deal, if they're a big deal to you, it's because you're trusting a system with a single point of failure, and the con…

Phone failing is a temporary inconvenience solved by getting a new one.

Backups failing mean 15 years of family photo memories forever lost.

Risk mitigations should reasonably scale in proportion to the cost of loss.

Re: In OpenZFS and Btrfs, everyone was just guessing

#154

Post by Too »

Lack of architectural direction is a common problem for many open source projects. They may have fancy user-facing documentation and a big test suite, but architecture schemas and rationale for design decisions are nowhere to be found.

Usually you have to dig through the git blame until you discover the commit message is nothing but “Merge fix-something into main”, from there you have to crawl the commits to find a PR number. Inside the PR, the only useful data is “Fixes: #1234”. From there, if you are lucky, you can find some rationale hidden among the 100 comments from stale-bots. If you are not lucky, the info you need is described in another issue of another library that is cross-linking to this issue.

For something sensitive as a file system, hearing they are no better is extra scary.

Re: In OpenZFS and Btrfs, everyone was just guessing

#155

Post by rincebrain »

rincebrain wrote: post #59
rincebrain wrote: post #56

Earlier quoted context omitted.

First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris. _Some_ of the Linux systems had a problem where…

A second fun one. In a previous life, I was paid to build storage systems for some HPC-ish workloads. So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it. I qu…

One more for the road.

So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons.

The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus.

Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun.

One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked.

I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot.

The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features.

The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic.

But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.

Re: In OpenZFS and Btrfs, everyone was just guessing

#156

Post by mike_hock »

mischief6 wrote: post #152
GrumpySloth wrote: post #144

Earlier quoted context omitted.

> disabling the writing of 'last accessed' timestamps atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime.

while it is unfortunate it requires a write, it is useful in some cases. for example, finding old objects in a cache, or today i used it to find out what firmware my kernel is loading by checking atime of /lib/firmware.

So it's useful as a debugging feature that you might want to briefly turn on once every decade.

Finding old objects in a cache is an issue specific to caches and should be implemented there.

Re: In OpenZFS and Btrfs, everyone was just guessing

#157

Post by eternityforest »

3np wrote: post #55
RantyDave wrote: post #37

Earlier quoted context omitted.

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…

> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for…

Data loss bugs in ZFS?? Wow.

My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.

Re: In OpenZFS and Btrfs, everyone was just guessing

#158

Post by cromka »

mischief6 wrote: post #152
GrumpySloth wrote: post #144

Earlier quoted context omitted.

> disabling the writing of 'last accessed' timestamps atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime.

while it is unfortunate it requires a write, it is useful in some cases. for example, finding old objects in a cache, or today i used it to find out what firmware my kernel is loading by checking atime of /lib/firmware.

For the latter you could enable it, reboot and get the same information right away, then disable it?

Re: In OpenZFS and Btrfs, everyone was just guessing

#159

Post by mnd999 »

jeffbee wrote: post #120
frankjr wrote: post #45

Earlier quoted context omitted.

A more "correct" fix has been posted https://github.com/openzfs/zfs/pull/15615

Neither of these fixes are great because neither of them include tests. This is a dataloss bug in a filesystem, no existing test covered it, and no new tests were added to demonstrate the efficacy of the supposed fix or prevent regressions.

The same was true of the fix for the completely deterministic cp from unencrypted to encrypted with block cloning. There doesn’t seem to me much incentive to write tests.

Re: In OpenZFS and Btrfs, everyone was just guessing

#160

Post by 3np »

eternityforest wrote: post #157
3np wrote: post #55

Earlier quoted context omitted.

> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for…

Data loss bugs in ZFS?? Wow. My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.

Comparing zfs/btrfs with ext4 is not entirely fair. They're in different domains when it comes to scope and features. To some extent, silent corruption in ext4 is expected but in zfs it's a bug.
Post reply on HN