Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In OpenZFS and Btrfs, everyone was just guessing

phoronix.com

Post reply on HN
Print view View original

81–90 of 172 posts

Re: In OpenZFS and Btrfs, everyone was just guessing

#81

Post by erhaetherth »

colechristensen wrote: post #76
3np wrote: post #55

Earlier quoted context omitted.

> whether or not the filesystem attempts deduplication Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case. > So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference. 100%. So much depends on the particular workload and resources. > And making backups, even for…

Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup. It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC als…

Oh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.

Re: In OpenZFS and Btrfs, everyone was just guessing

#82

Post by hypothesis »

phkahler wrote: post #66
vardump wrote: post #43

Earlier quoted context omitted.

Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

>> When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring. I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speed…

I wonder if anyone else instantly gets a visual after reading your story:

“Business woman on plane: Are there a lot of these kinds of accidents?

Narrator: You wouldn't believe.

Business woman on plane: Which car company do you work for?

Narrator: A major one.”

Re: In OpenZFS and Btrfs, everyone was just guessing

#83

Post by onetimeuse92304 »

rcxdude wrote: post #52
mghardal wrote: post #38

Earlier quoted context omitted.

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

from the description the flash was on a separate chip: I would guess most likely a SPI interface. You can absolutely get noise picked up on a SPI interface which looks like a command: and if an erase command is 16 bits long, you are going to see that command just with random bits every so often.

Yes, it was separate chip (per description). Yes, it was SPI flash. And yes, the word was 16 bits.

Re: In OpenZFS and Btrfs, everyone was just guessing

#84

Post by onetimeuse92304 »

calamari4065 wrote: post #47
mghardal wrote: post #38

Earlier quoted context omitted.

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

>The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.

We had to clean out a small warehouse and repurpose it for testing rig. We have created a piece of hardware to do automated testing and recording, and then we created a small production chain to and produced 100 of them.

Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).

Re: In OpenZFS and Btrfs, everyone was just guessing

#85

Post by onetimeuse92304 »

vardump wrote: post #43
onetimeuse92304 wrote: post #25

Earlier quoted context omitted.

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

Another source of bogus commands can be brownouts. When the voltage is low enough to confuse communication signals, but not low enough to prevent destructive events from occurring.

All the stuff that counted on our board had brownout protection. And if it didn't it is relatively easy to deal with (as long as you are aware of the problem).

(We may have learned a bit about noise and improved further iterations of the device, but sw change was enough.)

Re: In OpenZFS and Btrfs, everyone was just guessing

#87

Post by onetimeuse92304 »

mghardal wrote: post #38
onetimeuse92304 wrote: post #25

Earlier quoted context omitted.

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

Which mcu are you using ? The possibility of noise becoming a flash command is practically 0. Also too much noise will more importantly corrupt you ram. Flash can be sensitive to noise and that can cause bit flips.

1. What does choice of MCU have to do with it?

2. Noise can definitely affect other components. Traces are antennas that both broadcast as well as receive signals.

As technology marches forward, the voltages and tolerances for communication become smaller and tighter. At the same time, you integrate more functionality on the board so maybe your sensitive MCU and flash and ram are all living close by to some radio equipment and PSU and whatever else. Power interruptions to certain components can create momentary flashes of noise that can act as EMP bomb going off. On other occasions a noisy, badly designed high frequency power component can emit noise that is picked up by some random traces due to their geometric configuration and length.

Your comment suggests you never worked on a large project with thousands of components and dozens of major chips?

Re: In OpenZFS and Btrfs, everyone was just guessing

#88

Post by onetimeuse92304 »

onetimeuse92304 wrote: post #25
nabla9 wrote: post #3

> Everyone, literally, is just guessing. And then running scripts to evaluate the odds that things are actually working correctly. Just like BTRFS. When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening. In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once…

The hardest bug I have ever worked with was an embedded device losing all data on the flash chip. But the issue was, this was happening even after we removed all instructions to delete data from the flash. The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness…

I forgot to mention the best part of it. The fix was a one character code change (to enable a feature that we have disabled before).

Re: In OpenZFS and Btrfs, everyone was just guessing

#89

Post by tibbydudeza »

The crappiest bug I had was at one store - instead of using the recommended kit they wanted a cheaper solution so opted for a different barcode scanner but still reputable from a reputable supplier.

Store started complaining about sometimes scanning muffins - this was where the price was embedded inside the barcode - the prices were ridiculous for muffins but the barcode has a luhn check digit.

Must be our code management said - so we looked at our serial driver until we eventually found evidence through logging all serial I/O traffic that it was the barcode scanner that was sending back a corrupted barcode.

I would more worried what you can't see - SSD controllers - yikes don't want to think about the bugs in those things.

Post reply on HN