How we spent two weeks hunting an NFS bug in the Linux kernel

Just to give a picture how similar situations play out in closed source environments: They would have a key account manager who's live they would turn into hell. And depending on the size of gitlab's business in relation to the customers of the software provider it would have more or less effect towards software development. Reaction might range from having a fix in 2 days plus on-site engineer/consultant visits plus inviting the corresponding manager of gitlab to dinner for $100+/person to not even getting a response to even getting an angry call from the provider side's manager that the account will be closed if gitlab doesn't behave.

So, Github owned by Microsoft might not even notice such a bug for long, since the engineer would be able to get it fixed through an email or two. While the pre-Github-purchase Gitlab might have gotten no fix at all.

In that regard one might argue that open source doesn't completely destroy the ability to solve problems, but open source certainly helps balancing out the odds for different competitors.

It depends.

Firstly NFS only really works reliably when your network has harmonised UIG/GIDs. Thats the first pain point. This normally means LDAP/AD or shipping /etc/passwd (_shudders_) Also you need to squash root, otherwise people who are local root can do lots of naughty things.

Then you have to make sure that your mountpoint doesn't go away, because stale file handles are a pain in the arse.

Then you have file locking, which causes loads of other pain aswell. Most people turn that off.

After that its mostly alright.

nfsv4 has certain things that are good (pNFS, kerberos, etc) but support was not that great.

Totally disagree. You have to understand how NFS caching effects your application and what kind of performance and security/availability you need. At that point you can make NFS work for you. You also have to make good decisions about how you implement. This type of use case (long open file waits with concurrent access) are a nightmare with any shared filesystem. Most approaches dealing with possibly stale content are shoulder shrugs.

At GitLab we really care about our culture and core values [1]. As freddie said below: "Start doing it, celebrate it, reward it". If you are not sure how to make the first move, I would like to say that transparency is what really pushes everyone forward.

Start iterating on transparency, it may be hard but you will see the great results and it will make everyone around collaborate much more.

You probably heard of the event [2] which occurred almost 2 years ago - people are still talking about it and we are really happy and impressed to see everyone, including us, learning from that experience.

I hope this non-technical suggestion will help you to think about a solution to your question. If your team is not used to this kind of openness, eventually they will like the positive feedback from the community (we see that as a small iteration :-)). A comment section at [2] may be the extra source of motivation.

Have a nice day,

Djordje - Community Advocate at GitLab

[1] https://about.gitlab.com/handbook/values/#transparency

[2] https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-...

Really nice. I have more respect for GitLab now. That's a great write-up, and it led me to read some of their other nice reports too.

It's not exactly new for NFS to have cache coherency "surprises". But it should have "close-to-open" coherency at least, and the bug found by GitLab fails even that.

Here's an anecdote.

A Mac client talking to Samba on Linux. The client deletes random files that the client isn't even looking at, but which happen to be changed on the server around the time the client looks at the directory containing those files.

I am not joking. Randomly deleting files it's not even reading.

It delayed a product rollout for about 8 months. I was sure there must be a flaw in some file-updating code, somewhere in application code running on Linux. What else would make update-by-rename-over files disappear once every few weeks? Surely the usual tmpfile-fsync-rename dance was durable on Linux, on ext4? It must have been a silly, embarrasing error in the application code right? Calling unlink() with the wrong string or something.

But no, application was fine. Libraries were fine. And the awful bugs in VMware Fusion's file sharing were not to blame this time. (Ahem, another anecdote...)

It only happened every few weeks. A random file would disappear and be noticed. A web application would be told to update a file, and it'd spontaneously complain that the file was gone. It wasn't reproducible until we went all-out on trying to make it happen more often. But they kept disappearing.

Things like invoices data files and edited documents. Once every few weeks for no obvious reason. Not happy. And not safe to deploy.

Eventually, we found a very old bug in Emacs which deletes the file that's being saved in rare circumstances that only manifest when file attributes change at the wrong moment, which does happen with the weird and wonderful Mac SMB client's way of caching attributes. We thought we'd found the cause with great relief, and could proceed to rollout. Until after a few weeks, another file disappeared. No!

It took weeks of tracing, reproducing, and learning new debugging tools (like auditd running permanently) to rule out faults in (1) the application code and libraries, (2) Linux itself, (3) Samba, (4) tools used on the Mac when viewing a directory, and viewing and editing files.

Nope, it wasn't a bug in application code after all. There weren't any faulty calls or wrong strings. Logging would have caught them. Linux rename() was fine, not to blame. It wasn't a durability problem on power loss (the reason you need fsync with rename). Nor VMware disk image snapshots, even though other bugs were spotted with those. Nor was it the Emacs bug although that was a surprise to find.

The reproducer turned out to be "run cat a lot on the Mac, on a file which isn't being changed at all, while repeatedly updating another file on Linux in the same directory, using rename to update. Watch the updated file disappear eventually".

auditd showed Samba was doing the deletes, so I suspected a crazy bug in Samba and had to work quite hard to convince myself Samba was only doing what it was told by the client. I hoped it was Samba, because that's open source and I can fix that.

No, it was an astonishingly crappy bug called "delete random files once in a blue moon, hahaha!" in the Mac SMB client, which happened to occasionally be used to look in the same directory, which happened to be shared over Samba for convenience to look at it.

The confirmation of cause was from watching the SMB protocol, looking at Samba logs set to maximum verbosity, and lots of reading.

atq2119 says: "Imagine this same scenario if GitLab was using a closed source operating system. Would they have been able to track this down?"

I think I've had an experience like that - the above bug in the Mac SMB client. (Seriously, deleting random files.)

Googling reveals similar-sounding bugs at least two versions of OSX later. Yuck. I have no idea how to meaningfully get these things fixed or usefully reported. And I've had enough to stop caring anyway. The workaround is "force it to use SMB v1" (ye olde anciente). I can imagine the cause is something trivial in directory caching; it's probably just a few lines to fix.

I'm certain if the Linux client had a bug like that, it would be fixed very quickly, and probably backported by the big distros. I'm certain a Linux SMBFS developer would have been very helpful. And, there's a fairly good chance I could have fixed it myself and submitted the patch - probably less work than finding the cause, in this instance.

As it is, I don't think I could have found the culprit if I couldn't look at the Samba source to understand in detail what was going on in the SMB network protocol, or if I didn't have excellent tracing tools in Linux to find which process was responsible for stray deletions (i.e. not my application code, but Samba, which was doing as requested).