I always wonder why Linux doesn't seem to have any kind of tests. How can they afford not to have regression tests for bugs they fixed? How do they know that this bug fix didn't break anything? What does "never break userspace" even mean if there is no way to check whether userspace has been broken?
How we spent two weeks hunting an NFS bug in the Linux kernel
51–60 of 69 posts
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#52I always wonder why Linux doesn't seem to have any kind of tests. How can they afford not to have regression tests for bugs they fixed? How do they know that this bug fix didn't break anything? What does "never break userspace" even mean if there is no way to check whether userspace has been broken?
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#53The Red Hat BZ is "Access Denied", so can't see if it's fixed in RHEL & CentOS yet: https://bugzilla.redhat.com/show_bug.cgi?id=1648482 :(
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#54Earlier quoted context omitted.
You should name your university. So others can avoid it. Because not all are like that.
i dont believe there is sense in naming it so people can scape goat it anf solve things by avoiding one bad university. because i think the BA/MA system is a road paved with good intentions. but its leading us to hell. naming any singular entity would just make us think its them to blame, and i believe the problem is endemic. i sat once on a table with PHD students and complained about the quality of introductory cou…
This is what universities have always believed that they are for. PhD courses in particular. There used to be a separate category of school that was both technical and employment focused; in the UK these were called "polytechnics", in the US they would be things like the Texas Agricultural and Mining College. For complex reasons due to both the class system and the set of accidents of history that caused a lot of startup founders to come from places like Stamford, they have become a "unfashionable".
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#55A tangential question, the post links to an earlier post[1] saying that GitLab itself doesn't use NFS anymore, pointing out that they migrated to Gitaly. But ultimately Gitaly will need to do a local FS operation, so there's still the problem of ensuring HA for a given repository. GitHub solved this by writing their own replication layer on top of Git[2], but what's GitLab doing? Manually sharding repos on local FS's…
We are working on Gitaly HA. You can check out the Epic here: https://gitlab.com/groups/gitlab-org/-/epics/289
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#56The recommended way to perform atomic writes on POSIX is the create-write-fsync-rename-fsyncdir[0] dance. But that replaces the original file which causes ESTALE for all readers on NFS servers that don't support "delete on last close"[1] semantics.
This breaks common pattern where you can continue reading slightly stale data from unlinked files while writers updating the data atomically. In other words it makes it much harder to do filesystem concurrency correctly which already is hard enough.
A practical case where I'm seeing it is on Amazon's EFS. Updating thumbnails occasionally results in torn images because the server tries to send a stale file.
[0] https://danluu.com/file-consistency/ [1] http://nfs.sourceforge.net/#faq_d2
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#57Earlier quoted context omitted.
This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it. (I got as far as disassembling their DLLs to point to the exact problem)
Yeah, I don't really buy the parent comment's insinuation that Microsoft GitHub would be more likely to fix it than GitHub previously. My company has shipped hundreds of thousands of devices with a Microsoft OS on it, and they virtually never fixed anything reported. At least one bug was very serious, and we thought they would have to fix it, but it just didn't happen. What's hundreds of thousands to the hundreds of…
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#58Earlier quoted context omitted.
NFS requires long admin experience and tuning for each use case. The only comment I can agree with you on is that the linux automounter is lackluster. We used BSD amd for many years with good success.
> NFS requires long admin experience and tuning for each use case. Depends on what you're going to do with it. For something like sharing home directories, it works well enough. The defaults are usually pretty decent. There's unfortunately a lot of obsolete NFS tuning advice hanging around on the internet that seems to get cargo culted over and over again. Like the advice to set some specific rsize/wsize settings bec…
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#59Earlier quoted context omitted.
> NFS requires long admin experience and tuning for each use case. Depends on what you're going to do with it. For something like sharing home directories, it works well enough. The defaults are usually pretty decent. There's unfortunately a lot of obsolete NFS tuning advice hanging around on the internet that seems to get cargo culted over and over again. Like the advice to set some specific rsize/wsize settings bec…
Well enough isn't good enough for production. Having worked in high volume, highly available environments for years soho scenarios are not good examples. Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters) or pNFS and gluster require more than tuning mount options. Tuning NFS for a latency averse and throughput intensive application operating on large netcdf and hd…
How informative.
> Having worked in high volume, highly available environments for years soho scenarios are not good examples.
FWIW, I wasn't talking about SOHO. At least in my experience, defaults work well for home & shared work dirs for O(10k) users (not all simultaneously active, though). HA is a pain, though, if you want to DIY, I'll grant you that.
> Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters)
Complex? Sounds like a pretty standard NFS environment.
> or pNFS and gluster require more than tuning mount options.
Yeah, no personal experience there. What did you have to do there?
We did have a clustered NFS appliance for HPC use a decade or so ago. People like to complain how Lustre is a beast to run, but IME Lustre has been smooth sailing compared to the grief that POS gave us. But that wasn't really the fault of the NFS protocol per se, it was just the architecture as well as the implementation of that appliance was crap, particularly so for HPC.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#60Earlier quoted context omitted.
This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it. (I got as far as disassembling their DLLs to point to the exact problem)
Yeah, I don't really buy the parent comment's insinuation that Microsoft GitHub would be more likely to fix it than GitHub previously. My company has shipped hundreds of thousands of devices with a Microsoft OS on it, and they virtually never fixed anything reported. At least one bug was very serious, and we thought they would have to fix it, but it just didn't happen. What's hundreds of thousands to the hundreds of…
If Linux would be a proprietary product, than post-acquisition-github would get a fix from Linux developers because Microsoft is big. Not Github would provide anything, but they would get something due to the size of the Microsoft empirial stamp.