Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

71–80 of 154 posts

Re: Silent Data Corruption Is Real

#71

Post by h2hn »

tnorgaard wrote: post #63
h2hn wrote: post #59

I started a really simple and effective project the last month to be able to fix from bitrot in linux(MacOs/Unix?). It's "almost done" just need more real testing and make the systemd service. I've been pretty busy the last weeks so I've only been able to improve the bitrot performance. https://github.com/liloman/heal-bitrots Unfortunatly, btrfs is not stable and zfs needs a "super computer" or at least as much GBs o…

Please stop spreading this misinformed statement. I assume you are referring to the ZFS ARC (Adaptive Replacement Cache). It works in much the same way as a regular Linux page cache. It does not take much more memory (if you disable prefetch) and will only use what is available/idle. We use Linux with ZFS on production systems with as low as 1GB memory. We stopped counting the times it has saved the day. :-) ECC is a…

Interesting.

That's rigth the kind of hardware I was referring to, 1 GB of plain RAM. Truly, I haven't tested ZFS yet for that reason I've always read that ZFS has big requirements so I refrained to try it. It seems I should give it a try. ;)

Btrfs is another story I've used it for years and I'd prefer not to have to use it anymore untill it'll become "stable" and "performance". :)

Re: Silent Data Corruption Is Real

#72

Post by Dylan16807 »

seiferteric wrote: post #15
nayuki wrote: post #6

Earlier quoted context omitted.

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da... https://www.spinics.net/lists/linux-btrfs/msg59190.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

Yes I have tried to use btrfs several times for work projects and personally because I was very excited about it but every time I have ran into severe bugs even though it was said to be "stable". I have given up for now, maybe I will check back in a couple more years.

I really like btrfs for my backup machine, but it sometimes manages to hang when it's cleaning up deleted snapshots. This is a problem that's much worse when on a hard drive or fragmented file system, but I've gotten it to happen even on a recently-balanced drive with a small amount of snapshots.

I recently had a server be crippled by running snapper on default settings for a few months. And after a couple days of balancing (which desperately needs a throttle control) it wasn't much better, so I gave up on having it run btrfs.

I think the record I managed was slightly over two minutes of btrfs blocking all disk I/O. Something is deeply wrong with how it organizes transactions.

Re: Silent Data Corruption Is Real

#73

Post by gbrown_ »

zzzcpan wrote: post #34
elfchief wrote: post #29

Earlier quoted context omitted.

If nothing else, you can log the error. If you have RAID1 you can recopy the block from a good copy. It is, honestly, probably situation-specific, but step #1 is always going to be "identify that you have a problem"

That's the thing, identifying the problem on the filesystem level is useless if you can only correct it on another level. Unless your filesystem is distributed and self-healing, it's not a place for checksums, it must remain a thin predictable layer on top of a disk.

> That's the thing, identifying the problem on the filesystem level is useless if you can only correct it on another level.

Nope I'd take a system that can tell me "Hey sorry I don't have the right data" over "Here's some data, looks good to me" any day of the week.

Also as ProblemFactory points out, ZFS will self-heal whenever possible.

Re: Silent Data Corruption Is Real

#74

Post by Dylan16807 »

gjjrfcbugxbhf wrote: post #65
chungy wrote: post #55

Earlier quoted context omitted.

Short answer: No it wouldn't make it easier to port. Longer answer: The Linux subsystem in Windows 10 only deals with userspace. It doesn't support kernel modules nor changes anything about making Windows drivers. Porting ZFS to Windows is certainly possible, but it will take quite a lot of effort, and the Linux subsystem is irrelevant in that situation.

Would a FUSE implementation of ZFS not be possible? (Just wondering I've no idea what's possible here)

A FUSE implementation of ZFS exists and works well, and adding FUSE support to the Windows 10 Linux subsystem appears to be reasonably high up on the priority list.

That doesn't get you access from Windows programs, but there are some other ways to do FUSE or FUSE-like things on Windows..

Re: Silent Data Corruption Is Real

#75

Post by rincebrain »

Dylan16807 wrote: post #74
gjjrfcbugxbhf wrote: post #65

Earlier quoted context omitted.

Would a FUSE implementation of ZFS not be possible? (Just wondering I've no idea what's possible here)

A FUSE implementation of ZFS exists and works well, and adding FUSE support to the Windows 10 Linux subsystem appears to be reasonably high up on the priority list. That doesn't get you access from Windows programs, but there are some other ways to do FUSE or FUSE-like things on Windows..

Last I knew, the zfs-fuse codebase hadn't been updated since before feature flags were added to any of the OpenZFS targets, so it's not a particularly well-supported solution...

Re: Silent Data Corruption Is Real

#76

Post by e12e »

drvdevd wrote: post #48
justincormack wrote: post #40

Earlier quoted context omitted.

ZFS on OSX has been revived I believe.

I wonder if the recent Linux syscall emulation on Windows would somehow make it possible or easier to port ZFS on Linux to Windows. I know you have the SPL anyway, so maybe with the addition of the Linux POSIX-ish layer in there this could be the case...

Fwiw I've successfully shared my luks-encrypted usb3 zfs-formatted disk to Windows pro on my Surface 4 via a hyper-v vm running Ubuntu and samba. It won't work for all external drives - you need to be able to set the drive as "offline" in device manager under Windows in order to pass it through to the hyper-v vm (and sadly this doesn't appear possible with the sdcard - I had hoped to install Ubuntu on the sdcard and have the option to boot from the sdcard and also boot into the same filsystem under hyper-v).

Re: Silent Data Corruption Is Real

#77

Post by ianhowson »

db48x wrote: post #53
dpedu wrote: post #38

Earlier quoted context omitted.

Yes it can. ZFS will only notice next time you scrub or read the sector.

Sure, but your ZFS pool will have redundancy, and ZFS will know which block was corrupted. This lets it recover from the error.

If the corruption occurred on disk, yes. If it occurred in memory then it will write multiple incorrect copies to disk.

Re: Silent Data Corruption Is Real

#78

Post by qb45 »

danjoc wrote: post #8

Closed source firmware on drives contain bugs that corrupt data. Are there any drives, available anywhere, that have open source firmware?

Buggy SSDs certainly are a thing, mainly due to the complexity of FTL, but if you have a story of FW corrupting data on spinning rust that would be news to me.

Re: Silent Data Corruption Is Real

#79

Post by tux1968 »

rleigh wrote: post #60
tux1968 wrote: post #51

Earlier quoted context omitted.

The license incompatibility has kept it out of the mainline kernel for Linux, so it's not really a viable option there in many situations. Linux is definitely lacking in this department.

That's a minor concern and unrelated to the portability claim. That comes down to choice, and is not a technical consideration. I'm using it with Ubuntu 16.04 LTS and 16.10 where it works out of the box. It's most certainly portable to and from Linux and other systems; I've done it personally, and it works a treat.

Minor for some, major for others. And it is technical too, because there are maintenance ramifications for it not being in the mainline kernel, for example how quickly a security patch can be applied.

And for others a constriction on which distribution they can move to etc. It just reduces the number of situations where it can be used, even if you find yourself in one where it can.

Re: Silent Data Corruption Is Real

#80

Post by X86BSD »

ianhowson wrote: post #77
db48x wrote: post #53

Earlier quoted context omitted.

Sure, but your ZFS pool will have redundancy, and ZFS will know which block was corrupted. This lets it recover from the error.

If the corruption occurred on disk, yes. If it occurred in memory then it will write multiple incorrect copies to disk.

This is why ECC is important. Many many people poo poo the idea that it's needed. But by not having it you have left a single vital part of the data path unprotected. And ram and disk is cheap, losing your data is not. The risk simply isn't worth it to save literally a few dollars.
Post reply on HN