How we spent two weeks hunting an NFS bug in the Linux kernel

i recently graduated and started work.

academic achievements, relating to actuall work, is a drop of piss in an ocean. my experience at a university actually made me lose respect for academics.

edit: who ever is downvoting is romanticizing the achievements of scientists of yore. or thinks that MIT is the norm.

no, for the most part its publish or die, ive heard professors refer to students as "harvest" and laugh, while copying slides off of google. ive seen professors lie their way into grants.

all that contrasted with how the industry actually works and its needs, and what it actually require the universities to produce.

yeah, ive become a achievement oriented cynic. titles truly only make me think less of a person if thats all they have to impress with.

In my experience, the quality of NFS client implementations varies significantly between different operating systems. We made _heavy_ use of NFS for home directories and application backing stores at the University where I used to work, and it was a very good experience -- but this was on Solaris 10 (and later, OpenSolaris) machines. We had heavy NFS client use on many multi-user machines (shell servers, Sun Ray servers, etc) and didn't see reliability problems. On the odd occasion that we needed to reboot the file server for updates, clients would pause and then resume promptly after the server rebooted.

Towards the end of my tenure there, I gave a Linux desktop a try. The NFS experience was amazingly bad by comparison; lots of issues with locking, with becoming disconnected (often until a reboot) from NFS servers, odd performance issues, reliability issues with the automounter, etc.

In the last few months I have tried the NFS client on my current Linux desktop again, thinking things might have improved -- they have, I guess, but not by much. It's still pretty easy for the client to get into a hung state if there's too much packet loss, or if the file server reboots, or whatever. I have to imagine that not enough people are really using Linux NFS clients in anger to drive fixing the issues with it. There is often no escape from the Quality Death Spiral.

It's amazing how much we take the power of free and open source software for granted these days.

Imagine this same scenario if GitLab was using a closed source operating system. Would they have been able to track this down? Quite unlikely, but maybe if they were even more persistent and got lucky. Would they have been able to fix it? Absolutely not. They'd be at the mercy of the vendor.

I think NFS requires to have some knowledge for how you want to use it. The defaults seem reasonable, but I also get confused with all the v3/v4 differences etc. I use it at home to connect everything to my NAS and it works very well for me. My export is this:

  /mnt 192.168.8.0/24(rw,sync,insecure,no_subtree_check,crossmnt,all_squash,anonuid=0,anongid=100)

and my client config with autofs is this:

  rxd01 -fstype=nfs4,ro,soft,noatime,nodiratime,intr,rsize=65536,wsize=65536,nosuid,tcp,allow_other 192.168.8.3:/mnt/rxd01

If you know what each option does, how your network is setup and what your server/clients are capable of, you will eventually find the right settings, but it's not a good OOB experience.

It's an extremely old design, designed primarily for read-sharing and with its concurrency features retrofitted. It worked absolutely fine for network-booting diskless Sun workstations in the 90s but its failure modes are just too annoying for modern usage.

Trying to do anything like a database (and the 'git gc' process described is exactly that, a tiny database) over NFS requires the use of very specific techniques to get right.

Sibling commentator has it right - for unreliable WAN networks S3 offers far better semantics, because it's not quite a filesystem.