So par for the course for about every piece of complex enough software on this planet.
In OpenZFS and Btrfs, everyone was just guessing
101–110 of 172 posts
Re: In OpenZFS and Btrfs, everyone was just guessing
#102I don't know about BTRFS, but with ZFS there's a lot of bad information and assumptions floating around. It's one of my favorite topics to ask AI about because it shows the obvious deficiencies in AI as it regurgitates all the bad info. So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing somethi…
On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more aki…
Also, compression. CPUs are more than capable of handling it on the fly, and the reduced dataset size helps IO.
This [0] is a brief post on MySQL, but most of it applies equally to Postgres.
[0]: https://www.percona.com/blog/mysql-zfs-performance-update/
Re: In OpenZFS and Btrfs, everyone was just guessing
#103Earlier quoted context omitted.
One of my favorite comments chains here was was something to the affect of: Commenter posts a nice summary of how ZFS will try a quick compression algorithm to see if a block is even compressible and then try other more expensive ones to achieve better compression. Another commenter asks how they know that. First commenter replies that they wrote the code for it. Nothing more than a ZFS user (to some extent my compan…
You probably mean [1]. [1] - https://news.ycombinator.com/item?id=36608024
That is it, thank you.
Re: In OpenZFS and Btrfs, everyone was just guessing
#104> Good god people, this is not the way something as complex as a file system should be developed. There need to be lead architects that define the operation of the system, document it clearly and concisely, and create verification systems using both coverage and fuzz testing. And as new changes and/or features are introduced the verification systems must be updated to accommodate them.
ZFS has literally all of these things. I’m not sure if he’s just new to the community or making really poor assumptions based on his experience with btrfs.
Heck just look at GitHub. Anytime a new feature is being developed part of the merge process includes results from the test suites…
Re: In OpenZFS and Btrfs, everyone was just guessing
#105Earlier quoted context omitted.
>> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring. I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speed…
I wonder if anyone else instantly gets a visual after reading your story: “Business woman on plane: Are there a lot of these kinds of accidents? Narrator: You wouldn't believe. Business woman on plane: Which car company do you work for? Narrator: A major one.”
I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.
Re: In OpenZFS and Btrfs, everyone was just guessing
#106Earlier quoted context omitted.
> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for…
Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC als…
Deduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.
Re: In OpenZFS and Btrfs, everyone was just guessing
#107> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…
The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…
epic debugging story
Re: In OpenZFS and Btrfs, everyone was just guessing
#108> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…
I never really accept bug resolutions like that. Oracle corrupted data because a network driver fed it bad information? Did this network driver manage to preserve the legality of the corrupted info? Unlikely, so why did oracle believe it? If the corrupted message was still legal, then it sounds like the protocol isn't robust. I'm sure there's all kinds of nuance to your tale, but it still sounds like Oracle's bug to…
network drivers can usually corrupt arbitrary kernel memory
Re: In OpenZFS and Btrfs, everyone was just guessing
#109> Good god people, this is not the way something as complex as a file system should be developed. There need to be lead architects that define the operation of the system, document it clearly and concisely, and create verification systems using both coverage and fuzz testing. And as new changes and/or features are introduced the verification systems must be updated to accommodate them. ZFS has literally all of these…
It's the phoronix-forum...90% troll's and wannabe OS-engineers.
Re: In OpenZFS and Btrfs, everyone was just guessing
#110Earlier quoted context omitted.
I wonder if anyone else instantly gets a visual after reading your story: “Business woman on plane: Are there a lot of these kinds of accidents? Narrator: You wouldn't believe. Business woman on plane: Which car company do you work for? Narrator: A major one.”
I heard the story decades ago, and the problem had been fixed some years prior to that. There are undoubtedly hundreds of similar issues across the tech industry, most get quietly fixed before much damage is done. I only tell the story because it's an example of how obscure details come together in unexpected ways to cause failures. That happens everywhere.
And how brittle the borders we programmers see a strict, as a branching condition, "this should never happen". But in hardware, things are different.
How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?