Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

51–60 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#51

Post by YetAnotherNick »

vardump wrote: post #42
nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

Network drivers contribute to "bugs" surprisingly often. I've had my share of issues with them. I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.

How can network driver corrupt a database?

Re: In OpenZFS and Btrfs, everyone was just guessing

#52

Post by rcxdude »

mghardal wrote: post #38
onetimeuse92304 wrote: post #25

Earlier quoted context omitted.

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.

Re: In OpenZFS and Btrfs, everyone was just guessing

#53

Post by [deleted] »

onetimeuse92304 wrote: post #25
nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

[deleted]

Re: In OpenZFS and Btrfs, everyone was just guessing

#54

Post by rincebrain »

hotpotamus wrote: post #32
donmcronald wrote: post #18

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing somethi…

One of my favorite comments chains here was was something to the affect of: Commenter posts a nice summary of how ZFS will try a quick compression algorithm to see if a block is even compressible and then try other more expensive ones to achieve better compression. Another commenter asks how they know that. First commenter replies that they wrote the code for it. Nothing more than a ZFS user (to some extent my compan…

You probably mean [1].

[1] - https://news.ycombinator.com/item?id=36608024

Re: In OpenZFS and Btrfs, everyone was just guessing

#55

Post by 3np »

RantyDave wrote: post #37
donmcronald wrote: post #18

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing somethi…

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…

> whether or not the filesystem attempts deduplication

Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case.

> So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.

100%. So much depends on the particular workload and resources.

> And making backups, even for ZFS backed filesystems. Sacrilege, I know.

Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.

Re: In OpenZFS and Btrfs, everyone was just guessing

#56

Post by rincebrain »

nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

First, you may enjoy [1].

Second, a story from many years ago.

The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris.

_Some_ of the Linux systems had a problem where eventually, the NIC would stop responding and dmesg would flood with "Tx Unit Hang" or "Rx Unit Hang", and not work again until a reboot.

If we swapped the NIC from a system where this never happened to one with the same model NIC where this happened, the problem kept happening - that is, it was something about the system+NIC, not just the NIC.

Intel eventually gave us a shipping label to borrow one of our systems when they said they hadn't been able to reproduce it but kept having people report it, and we said "we have a number of them".

They made some really exciting noises after some debugging, and eventually concluded the issue was that our really cheapass Sempron board that we sent them screwed up badly if you tried negotiating 64-bit PCI in their normal PCI slots, and would sometimes miss the messages between the NIC and the host saying that Tx Ring buffer 0 is free/full, etc, and eventually, it would miss so many of those that it would think the NIC had no more buffers, and here we are. They couldn't reproduce it because all their testbeds were well-designed and tested things.

Choice quotes mid-debugging include "I do not see how anyone could make a PCI device work reliably in this system".

They did figure out a workaround, and ship it, and asked if they could keep the system permanently for future testing.

But that was exciting.

[1] - https://github.com/openzfs/zfs/pull/15588

Re: In OpenZFS and Btrfs, everyone was just guessing

#57

Post by akira2501 »

nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

> certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.

Found a similar issue in a switch stack relating to multicast stream delivery. Knowing about this class of bug and having isolated it down to the switch, as the only possible device in the path, being responsible, I filed a ticket with the appropriate team to get the stack rebooted.

They were understandably skeptical of my supposed prescience and so I had to spend a considerably larger amount of time writing test cases and running deep packet captures in order to prove that a single bit, in the port field, was getting flipped under circumstances that were repeatable but not entirely explainable without even more work.

Finally, they relented and agreed that, just turning it off, then on again, was indeed the best initial solution. Fortunately with all the test harnesses in place we were able to prove that it cleared the issue on that set of ports.

Re: In OpenZFS and Btrfs, everyone was just guessing

#58

Post by throwaway2037 »

YetAnotherNick wrote: post #51
vardump wrote: post #42

Earlier quoted context omitted.

Network drivers contribute to "bugs" surprisingly often. I've had my share of issues with them. I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.

How can network driver corrupt a database?

By corrupting the data that was received.

Re: In OpenZFS and Btrfs, everyone was just guessing

#59

Post by rincebrain »

rincebrain wrote: post #56
nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

First, you may enjoy [1]. Second, a story from many years ago. The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris. _Some_ of the Linux systems had a problem where…

A second fun one.

In a previous life, I was paid to build storage systems for some HPC-ish workloads.

So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it.

I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them.

Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired...

This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing.

Huh.

So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it.

So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up.

The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that.

I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.

Re: In OpenZFS and Btrfs, everyone was just guessing

#60

Post by cesarb »

favorited wrote: post #40
nubinetwork wrote: post #4

So many hate mongers in that thread, rubbing salt in the wound rather than trying to fix a problem that could happen for any filesystem.

People are unreasonably tribal about file systems.

> People are unreasonably tribal about file systems.

I think it's probably worse than usual with ZFS, mostly for two reasons: a lot of hype (justified or not) early on its life, leading to hype aversion and/or backlash, and a "sour grapes" effect because its license ensures it will never (or at least until its copyright expires) be merged into mainline Linux, so those who can't or won't use it (due to a need or desire to run only mainline Linux kernel code) feel left out and/or get annoyed at seeing it pushed as the best thing since sliced bread all the time.

Adding to that, its main competitor (btrfs) is known for having had (real or perceived) reliability issues early on its life, making its use more controversial (and its proponents more defensive at seeing the same reliability issues being raised all the time).

And file systems are important, since people are reasonably concerned about the integrity of their data. They are also not easy to migrate (other than a rarely used tool to migrate in place from the ext2/3/4 family to btrfs, migrating filesystems usually requires dumping and reloading the whole data), making the choice of filesystems an important decision.

Post reply on HN