Live data from Hacker News

How we spent two weeks hunting an NFS bug in the Linux kernel

about.gitlab.com

51–60 of 69 posts

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#51

I always wonder why Linux doesn't seem to have any kind of tests. How can they afford not to have regression tests for bugs they fixed? How do they know that this bug fix didn't break anything? What does "never break userspace" even mean if there is no way to check whether userspace has been broken?

Never break userspace means that if you break userspace, that's a bug. The effort spent on preventing bugs is another matter.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#52

I always wonder why Linux doesn't seem to have any kind of tests. How can they afford not to have regression tests for bugs they fixed? How do they know that this bug fix didn't break anything? What does "never break userspace" even mean if there is no way to check whether userspace has been broken?

https://stackoverflow.com/questions/3177338/how-is-the-linux...

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#53

The Red Hat BZ is "Access Denied", so can't see if it's fixed in RHEL & CentOS yet: https://bugzilla.redhat.com/show_bug.cgi?id=1648482 :(

You can sign up for an account to see the bug report status. The patch has not yet been backported.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#54
post #34

Earlier quoted context omitted.

You should name your university. So others can avoid it. Because not all are like that.

i dont believe there is sense in naming it so people can scape goat it anf solve things by avoiding one bad university. because i think the BA/MA system is a road paved with good intentions. but its leading us to hell. naming any singular entity would just make us think its them to blame, and i believe the problem is endemic. i sat once on a table with PHD students and complained about the quality of introductory cou…

> "a university does not prepare for work! it prepares for research!"

This is what universities have always believed that they are for. PhD courses in particular. There used to be a separate category of school that was both technical and employment focused; in the UK these were called "polytechnics", in the US they would be things like the Texas Agricultural and Mining College. For complex reasons due to both the class system and the set of accidents of history that caused a lot of startup founders to come from places like Stamford, they have become a "unfashionable".

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#55
post #50
post #38

A tangential question, the post links to an earlier post[1] saying that GitLab itself doesn't use NFS anymore, pointing out that they migrated to Gitaly. But ultimately Gitaly will need to do a local FS operation, so there's still the problem of ensuring HA for a given repository. GitHub solved this by writing their own replication layer on top of Git[2], but what's GitLab doing? Manually sharding repos on local FS's…

We are working on Gitaly HA. You can check out the Epic here: https://gitlab.com/groups/gitlab-org/-/epics/289

So since redundancy & horizontal scaling are goals of Gitaly HA am I to understand that right now GitLab.com is run on some ad-hoc setup like what I described, and you can lose data if you're unlucky enough with a machine or two disks going down at the same time?

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#56
NFS open file handle semantics are quite an annoyance.

The recommended way to perform atomic writes on POSIX is the create-write-fsync-rename-fsyncdir[0] dance. But that replaces the original file which causes ESTALE for all readers on NFS servers that don't support "delete on last close"[1] semantics.

This breaks common pattern where you can continue reading slightly stale data from unlinked files while writers updating the data atomically. In other words it makes it much harder to do filesystem concurrency correctly which already is hard enough.

A practical case where I'm seeing it is on Amazon's EFS. Updating thumbnails occasionally results in torn images because the server tries to send a stale file.

[0] https://danluu.com/file-consistency/ [1] http://nfs.sourceforge.net/#faq_d2

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#57
post #44
post #41

Earlier quoted context omitted.

This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it. (I got as far as disassembling their DLLs to point to the exact problem)

Yeah, I don't really buy the parent comment's insinuation that Microsoft GitHub would be more likely to fix it than GitHub previously. My company has shipped hundreds of thousands of devices with a Microsoft OS on it, and they virtually never fixed anything reported. At least one bug was very serious, and we thought they would have to fix it, but it just didn't happen. What's hundreds of thousands to the hundreds of…

I think you misread the comment. Assuming that NFS was sold by NFS-Co, GitHub, as part of Microsoft, would be a big enough customer to get NFS-Co to fix the bug quickly. However, GitLab would be a tiny NFS-Co customer, and so the bug would have gone unfixed. The difference is in the size of who reports the bug.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#58
post #42

Earlier quoted context omitted.

NFS requires long admin experience and tuning for each use case. The only comment I can agree with you on is that the linux automounter is lackluster. We used BSD amd for many years with good success.

> NFS requires long admin experience and tuning for each use case. Depends on what you're going to do with it. For something like sharing home directories, it works well enough. The defaults are usually pretty decent. There's unfortunately a lot of obsolete NFS tuning advice hanging around on the internet that seems to get cargo culted over and over again. Like the advice to set some specific rsize/wsize settings bec…

Well enough isn't good enough for production. Having worked in high volume, highly available environments for years soho scenarios are not good examples. Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters) or pNFS and gluster require more than tuning mount options. Tuning NFS for a latency averse and throughput intensive application operating on large netcdf and hdf5 file hierarchies is a worthy example.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#59
post #42

Earlier quoted context omitted.

> NFS requires long admin experience and tuning for each use case. Depends on what you're going to do with it. For something like sharing home directories, it works well enough. The defaults are usually pretty decent. There's unfortunately a lot of obsolete NFS tuning advice hanging around on the internet that seems to get cargo culted over and over again. Like the advice to set some specific rsize/wsize settings bec…

Well enough isn't good enough for production. Having worked in high volume, highly available environments for years soho scenarios are not good examples. Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters) or pNFS and gluster require more than tuning mount options. Tuning NFS for a latency averse and throughput intensive application operating on large netcdf and hd…

> Well enough isn't good enough for production.

How informative.

> Having worked in high volume, highly available environments for years soho scenarios are not good examples.

FWIW, I wasn't talking about SOHO. At least in my experience, defaults work well for home & shared work dirs for O(10k) users (not all simultaneously active, though). HA is a pain, though, if you want to DIY, I'll grant you that.

> Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters)

Complex? Sounds like a pretty standard NFS environment.

> or pNFS and gluster require more than tuning mount options.

Yeah, no personal experience there. What did you have to do there?

We did have a clustered NFS appliance for HPC use a decade or so ago. People like to complain how Lustre is a beast to run, but IME Lustre has been smooth sailing compared to the grief that POS gave us. But that wasn't really the fault of the NFS protocol per se, it was just the architecture as well as the implementation of that appliance was crap, particularly so for HPC.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#60
post #44
post #41

Earlier quoted context omitted.

This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it. (I got as far as disassembling their DLLs to point to the exact problem)

Yeah, I don't really buy the parent comment's insinuation that Microsoft GitHub would be more likely to fix it than GitHub previously. My company has shipped hundreds of thousands of devices with a Microsoft OS on it, and they virtually never fixed anything reported. At least one bug was very serious, and we thought they would have to fix it, but it just didn't happen. What's hundreds of thousands to the hundreds of…

> I don't really buy the parent comment's insinuation that Microsoft GitHub would be more likely to fix it than GitHub previously

If Linux would be a proprietary product, than post-acquisition-github would get a fix from Linux developers because Microsoft is big. Not Github would provide anything, but they would get something due to the size of the Microsoft empirial stamp.

Post reply on HN