I'd wager money Brian Behlendorf has an idea how ZFS works..
In OpenZFS and Btrfs, everyone was just guessing
91–100 of 172 posts
Re: In OpenZFS and Btrfs, everyone was just guessing
#92I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing somethi…
On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…
For Postgres just follow the original doc:
https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
compression=lz4 recordsize=32K
In postgres -> full_page_writes=off
Re: In OpenZFS and Btrfs, everyone was just guessing
#93Earlier quoted context omitted.
Note that the bug that is the topic of discussion here predates OpenZFS. Whether or not there has been a slide in disciplined development in OpenZFS, this bug does not support that assertion.
Since the topic is whether or not the current OZFS developers understand what is going on well enough to reliably fix the bug, I think it still applies.
No, the topic of that thread is to prevent such bug's in the first place.
And i think that OP is right, development speed is really a bit too fast atm for a filestystem. Maybe a codereview ala openbsd could do the job?
Re: In OpenZFS and Btrfs, everyone was just guessing
#94First, there is a FreeBSD Errata Notice for this that offers an nice quick collection of the various bugs and subsequent repairs, with links to summaries, for anyone who is catching up on this issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308 Second, I don't like the editorialization of this title ("In OpenZFS and Btrfs, everyone was just guessing") at all. No, nobody was "just guessing", but as far as…
I just want to say that you can do checksums on...I mean under xfs/ext etc on linux with dm_integrity, I've talked to many admins who have never heard of this device mapper.
Re: In OpenZFS and Btrfs, everyone was just guessing
#95So many hate mongers in that thread, rubbing salt in the wound rather than trying to fix a problem that could happen for any filesystem.
People are unreasonably tribal about file systems.
Well, it's typical of the phoronix forum, they even start a fight over zstd compression levels. Like a bunch of 12 year olds who "mastered" installing system x, they now have to defend their choice of distribution, file system, desktop/wm, wayland or x11, and at the same time attack everything else because their choice was superior.
Re: In OpenZFS and Btrfs, everyone was just guessing
#96I see a lot of comments here using the terms ZFS and Open ZFS interchangeably.
But to my understanding, these are 2 different things.
- ZFS is the very stable, non open source project that only works on BSD. - Open ZFS is an effort to rewrite ZFS in a totally open and free format that now runs on Linux.
This was the idea I had, but I might be incorrect.
Re: In OpenZFS and Btrfs, everyone was just guessing
#97Somewhat related, RedHat withdrew from the attempt to productionize BRTFS, apparently it was fully removed in RHEL 8:
- https://access.redhat.com/solutions/197643
Also, there's a PDF doc (from a conference presentation), covering many layers of potential alternatives for ZFS features:
- https://news.ycombinator.com/item?id=38484598
But "just" slapping all these layers on each other (configuring each individually and then hoping for the best) doesn't seem like the best idea for production systems either.
I guess back in the Sun/Solaris days, people still had the luxury of releasing (major) features in major versions, every few years instead of weeks or months... and as far as storage, backup, security solutions go - can't really do this right with the move fast & break things culture...
Re: In OpenZFS and Btrfs, everyone was just guessing
#98Earlier quoted context omitted.
You're at odds with the desire (necessity, actually) to have precise low-level control, not just of the in-memory layout of the data structures, but also of the performance characteristics of your code (e.g. no unnecessary pointer-chasing). Higher-level languages tend to make it easier to pile up abstractions; we want the orthogonal property of making it harder to shoot yourself in the foot. You can look at ZFS for i…
I like Rust, but you are going to be writing a lot of unsafe Rust for a filesystem implementation. Multiple processes are writing to the filesystem simultaneously. "Ownership" is fuzzy and is moving around. At that point, is Rust buying you anything for how much it's going to get in your way? I really don't see an advantage to Rust when operating at these kinds of low levels.
"Just rewrite it in Rust" is a bit meme-y, but what I wanted to highlight is how you can look at ZFS itself for inspiration. ZFS has checksums to detect bit rot; maybe you want a language with better support for contracts / static analysis. ZFS has zvols to expose virtual block devices for use with other filesystems; maybe you want a language with safe C interop; etc. Any improvement in safety over plain old C would be desirable, and even unsafe Rust is safer.
Re: In OpenZFS and Btrfs, everyone was just guessing
#99I see a lot of comments here using the terms ZFS and Open ZFS interchangeably. But to my understanding, these are 2 different things. - ZFS is the very stable, non open source project that only works on BSD. - Open ZFS is an effort to rewrite ZFS in a totally open and free format that now runs on Linux. This was the idea I had, but I might be incorrect.
From my recollection, the basic story is Sun open sourced Solaris in the early 2000's (OpenSolaris). Today's OpenZFS codebase is an incarnation of the code released in OpenSolaris. It has since been ported to Linux and continues on as a separate thing from ZFS. Oracle has since "un-open sourced" Solaris, and is why ZFS and OpenZFS are different things today, although they both share a heritage.
Re: In OpenZFS and Btrfs, everyone was just guessing
#100> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…
I never really accept bug resolutions like that. Oracle corrupted data because a network driver fed it bad information? Did this network driver manage to preserve the legality of the corrupted info? Unlikely, so why did oracle believe it? If the corrupted message was still legal, then it sounds like the protocol isn't robust.
I'm sure there's all kinds of nuance to your tale, but it still sounds like Oracle's bug to me. They just found what it was in your system that introduced the behaviour they hadn't anticipated, and then got you to remove it.