Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS on Linux: Unlistable and disappearing files

github.com

Post reply on HN
Print view View original

141–150 of 168 posts

Re: ZFS on Linux: Unlistable and disappearing files

#141

Post by Annatar »

StaticRedux wrote: post #125
Annatar wrote: post #108

Earlier quoted context omitted.

Since when is introducing a major regression into something as critical as a filing system a hooray for open source? Only on GNU/Linux... and nobody thinks twice about it or bats an eyelash; instead, the scrambling to contain the regression is greeted with a hooray. It’s mentality like this which makes me want to never touch a computer again.

Introducing major bugs happens in software all the time. Finding it, working out a solution, and fixing it quickly is a hooray for open source.

How quickly a problem is understood and fixed has nothing whatsoever to do with open source. That is a fallacy.

Re: ZFS on Linux: Unlistable and disappearing files

#142

Post by ryao »

Annatar wrote: post #140
iforgotpassword wrote: post #135

Earlier quoted context omitted.

So what exactly is your suggestion? "lost+found is ugly so don't recover the files at all"? He said meta data is lost so I'm really curious to hear your elegant modern solution. This is a tool that might be needed on a few systems that suffered from this relatively short loved bug and then never again after that, so your solution better not require ten times the effort.

If it's a one time throwaway tool, then it doesn't need lost+found!

I was talking more about the idea of a lost+found directory. Brian suggested that we make it ‘.zfs/lost+found‘, which might be what we do. I am leaning toward extending scrub to add the function when a flag for an erratum check is passed. That implies that the tool would be a part of the driver so that it can be done online for minimal downtime.

Re: ZFS on Linux: Unlistable and disappearing files

#143

Post by [deleted] »

Annatar wrote: post #139
barrkel wrote: post #112

Earlier quoted context omitted.

Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?

I suggest completely backing out the code in question and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. metadata are not files so they have no business going into lost+found.

[deleted]

Re: ZFS on Linux: Unlistable and disappearing files

#144

Post by ryao »

Annatar wrote: post #139
barrkel wrote: post #112

Earlier quoted context omitted.

Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?

I suggest completely backing out the code in question and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest. metadata are not files so they have no business going into lost+found.

> I suggest completely backing out the code in question

This makes sense and we did that. It does not fix the fact that datasets exist where directories have incorrect sizes and files are orphaned because of the bad patch.

> and using a FreeBSD or an illumos based system like SmartOS. That's what I suggest.

While those are fine choices, they are not immune to bugs either. We had a space map corruption bug that we inherited from Illumos several years ago. We also inherited an issue in send/recv from illumos several years ago when hole_birth was introduced. FreeBSD’s TRIM implementation was far from perfect too and likely messed some things up given the long history of bug fixes that it required.

I do not mean to discourage people from using those systems, but it is silly to adopt them because a bug got past joint OpenZFS review and landed in the Linux port first. While they do a good job too, they are not infallible. The major OpenZFS platforms are roughly at parity in terms of their risk for bugs and regressions.

> metadata are not files so they have no business going into lost+found.

This makes no sense. How are orphaned files “metadata”? Why should they not be placed in a lost+found directory?

Re: ZFS on Linux: Unlistable and disappearing files

#145

Post by cookiecaper »

flukus wrote: post #15
zorkw4rg wrote: post #8

basically the reason I like my filesystems a few decades old and mature, I would not trust zfs or btrfs with anything critical.

Alternatively you can just avoid the bleeding edge versions of any file system.

Yes. I don't think anyone would contend that it's not bad for this to have landed in a stable release, but in fairness, that stable release was about 3 weeks old and ZoL is not yet 1.0. Mistakes like this are unfortunate but they do happen, and that's why many people wait several months before they incorporate new versions.

From the bug report, sounds like ZoL is integrating some new tests, learning how to retrieve orphaned files, and possibly strengthening its hashing mechanisms, so overall this will strengthen ZFS's robustness. Little consolation for those with damaged filesystems, but that's how the cookie crumbles sometimes.

Re: ZFS on Linux: Unlistable and disappearing files

#146

Post by [deleted] »

amelius wrote: post #88
dspillett wrote: post #86

Earlier quoted context omitted.

And that you regularly test that property.

But how do you test that reliably if the filesystem makes files disappear? I.e., you'll have an original that misses some files, and a backup that misses the same files. A diff will show nothing.

[deleted]

Re: ZFS on Linux: Unlistable and disappearing files

#147

Post by tinus_hn »

Annatar wrote: post #110
tinus_hn wrote: post #91

Earlier quoted context omitted.

Imagine if this had happened to Apple, everyone and his dog would be rolling over each other villifying them.

I paid good money to Apple so something like that would not happen, so yeah I’d be villifying them. You bet!

As opposed to the free RedHat Enterprise Linux operating system

Re: ZFS on Linux: Unlistable and disappearing files

#148

Post by ryao »

ryao wrote: post #9

We are working on it. We know what patch introduced the regression and 0.7.8 is going out soon to revert it. Until then, users should downgrade to 0.7.6 if they have not already. The Gentoo and EPEL maintainers have pulled the affected releases from the repositories (technically masked on Gentoo). Ubuntu was never affected. The regression makes it so that creating a new file could fail with ENOSPC after which files c…

I put an update here:

https://news.ycombinator.com/item?id=16806176

Re: ZFS on Linux: Unlistable and disappearing files

#149

Post by walrus01 »

rkagerer wrote: post #84
1024core wrote: post #57

Earlier quoted context omitted.

> also a good time to remind people that backups are not backups unless they're geographically diverse. ... and backups are not backups until you can reliably restore from them.

...and a DR plan isn't worth the paper it's written on unless you test it regularly (people leave, media gets corrupted... just like the military; do drills!)

... and a DR plan isn't worth anything unless you test the "people leave" part without any knowledge in their heads. your internal documentation and credential storage needs to be solid in case four people flying together all die abruptly in the same plane crash. the traditional "bus problem".

Re: ZFS on Linux: Unlistable and disappearing files

#150

Post by isnull »

arianvanp wrote: post #81
isnull wrote: post #77

Earlier quoted context omitted.

i have been using btrfs in production for years now and it has never failed me, and i am doing hundreds of snapshots and send/receiving, i've reconfigured raids on the fly and went from 6 disk raid 10 of mixed size to raid 10 of same size. we have in some cases had many power failures with no data loss. We also have some set up using md raid and some using hardware raid... when i hear people dogging btrfs it just spe…

Are you using btrfs native RAID support? The docs say it's really unstable so I was scared to use it

I run single with hardware or md raid, or btrfs raid 1 or 10. Usually the latter.

Raid5 and 6 are unstable in the "avoid at all costs" kind of way.

This may be of use if youd like to know more: https://btrfs.wiki.kernel.org/index.php/Status

Post reply on HN