Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

71–80 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#71

Post by linsomniac »

One thing to remember about OpenZFS is that it is in many ways a port of a file system more than the file system itself. I could see developers there being less steeped in the file system, and more familiar with the porting layer.

I'm going to spend a moment praising scripts though...

A dozen or more years ago I was running a bunch of backup servers. The backup system was rsync to a zfs, then snapshot the zfs. Then do a rolling delete of snapshots to achieve X daily, Y weekly, and Z monthly copies. This is because the hardlink trick is amazingly inefficient.

I started off using Nevada, but had issues and no real expertise there, so I switched to Linuz with ZFS fuse. That worked about as well as Nevada, but about once a month or two zfs-fuse would die.

I built a script that would simulate the access patterns of the backup systems. I then went through a few iterations with the developers of running the script (which logged its actions as shell commands, so the log could be re-executed), reporting the results to the developers, and them producing a fix. After a few back-and-forths with the developers, zfs-fuse became rock solid for my use case. And the script was the key.

Re: In OpenZFS and Btrfs, everyone was just guessing

#72

Post by duped »

bsder wrote: post #41
rollcat wrote: post #20

Earlier quoted context omitted.

You're at odds with the desire (necessity, actually) to have precise low-level control, not just of the in-memory layout of the data structures, but also of the performance characteristics of your code (e.g. no unnecessary pointer-chasing). Higher-level languages tend to make it easier to pile up abstractions; we want the orthogonal property of making it harder to shoot yourself in the foot. You can look at ZFS for i…

I like Rust, but you are going to be writing a lot of unsafe Rust for a filesystem implementation. Multiple processes are writing to the filesystem simultaneously. "Ownership" is fuzzy and is moving around. At that point, is Rust buying you anything for how much it's going to get in your way? I really don't see an advantage to Rust when operating at these kinds of low levels.

Have you ever written a file system? Most of the work is conforming to the semantics of the interface to the kernel/programs calling it. Very little of that requires unsafe code, and the bulk of the provably unsafe stuff (physically writing to memory/disk) is very simple.

The complex stuff can and should be written at a higher level than C.

Re: In OpenZFS and Btrfs, everyone was just guessing

#74

Post by chungy »

Wowfunhappy wrote: post #49

OpenZFS is on an exceedingly short list of software in my life that I actually trust to do what it's supposed to. Where most software just up and falls over every so often, ZFS chugs along, day in and day out. Then ZFS has one bug and everyone starts acting like the sky is falling. And sure, it was a bad bug, but you had to be pretty unlucky to trigger it, and it was present in ZFS for all of a month. Meanwhile, peop…

Block cloning happened to expose this bug easier, but it seems like this bug might actually date back to the very beginning of ZFS with Sun, with the opportunities to trigger it being so rare, that nobody had noticed until now.

I suppose it's still a bad bug, but it goes to show that file systems are complicated beasts, practically impossible to test all the ways they can be put through the wringer, and bugs like this can hide for 17 years before being unearthed.

Re: In OpenZFS and Btrfs, everyone was just guessing

#75

Post by HankB99 »

I'm a little sad to see this. I've been using ZFS on my PCs for years with minimal issue and significant benefits. [1] I've even contributed [2]. And I remain a huge fan.

I don't believe that "everyone is just guessing". There are some pretty knowledgeable folk that work on this.

[1] I've triggered some corruption in snapshots on an encrypted pool. No permanent problems resulted and no data was lost.

[2] I provided a very minor documentation fix that was encouraged and promptly merged. https://github.com/openzfs/openzfs-docs/pull/472

Re: In OpenZFS and Btrfs, everyone was just guessing

#76

Post by colechristensen »

3np wrote: post #55
RantyDave wrote: post #37

Earlier quoted context omitted.

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…

> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for…

Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup.

It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it.

In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.

Re: In OpenZFS and Btrfs, everyone was just guessing

#77

Post by chungy »

RantyDave wrote: post #37
donmcronald wrote: post #18

I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing somethi…

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…

> changing the page size to be the same size as Postgresql uses (8kb)

Mind, you shouldn't really change the record size without knowing your use-case. Make a PostgreSQL file system, set it's recordsize to 8k. That's a good idea.

Not so much a good idea doing that to all your filesystems. Stick to the defaults without a good justification to deviate.

Re: In OpenZFS and Btrfs, everyone was just guessing

#78

Post by MassPikeMike »

calamari4065 wrote: post #47
mghardal wrote: post #38

Earlier quoted context omitted.

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.

Once you have 700 units in the field, one failure per 700 days is no longer quite so rare.

Re: In OpenZFS and Btrfs, everyone was just guessing

#79

Post by eep_social »

calamari4065 wrote: post #47
mghardal wrote: post #38

Earlier quoted context omitted.

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.

And that’s running the particular op repeatedly, so in practice it’s an even rarer event. However with “millions of them in the wild” it doesn’t matter. Scaling is tough.

Re: In OpenZFS and Btrfs, everyone was just guessing

#80

Post by 3np »

colechristensen wrote: post #76
3np wrote: post #55

Earlier quoted context omitted.

> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for…

Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC als…

This is my understanding as well. These days we default to zstd, though.
Post reply on HN