Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent two weeks hunting an NFS bug in the Linux kernel

about.gitlab.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#41

Post by pjc50 »

erikb wrote: post #31
atq2119 wrote: post #26

It's amazing how much we take the power of free and open source software for granted these days. Imagine this same scenario if GitLab was using a closed source operating system. Would they have been able to track this down? Quite unlikely, but maybe if they were even more persistent and got lucky. Would they have been able to fix it? Absolutely not. They'd be at the mercy of the vendor.

Just to give a picture how similar situations play out in closed source environments: They would have a key account manager who's live they would turn into hell. And depending on the size of gitlab's business in relation to the customers of the software provider it would have more or less effect towards software development. Reaction might range from having a fix in 2 days plus on-site engineer/consultant visits plus…

This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it.

(I got as far as disassembling their DLLs to point to the exact problem)

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#42

Post by jabl »

romeisendcoming wrote: post #36
jclulow wrote: post #24

Earlier quoted context omitted.

In my experience, the quality of NFS client implementations varies significantly between different operating systems. We made _heavy_ use of NFS for home directories and application backing stores at the University where I used to work, and it was a very good experience -- but this was on Solaris 10 (and later, OpenSolaris) machines. We had heavy NFS client use on many multi-user machines (shell servers, Sun Ray serv…

NFS requires long admin experience and tuning for each use case. The only comment I can agree with you on is that the linux automounter is lackluster. We used BSD amd for many years with good success.

> NFS requires long admin experience and tuning for each use case.

Depends on what you're going to do with it. For something like sharing home directories, it works well enough.

The defaults are usually pretty decent. There's unfortunately a lot of obsolete NFS tuning advice hanging around on the internet that seems to get cargo culted over and over again.

Like the advice to set some specific rsize/wsize settings because the default is too small, oblivious to the fact that the NFS protocol allows the client and server to negotiate maximum sizes, and at least the Linux client and server have taken advantage of this negotiation mechanism for the past 2 decades or so.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#43

Post by virtualized »

I always wonder why Linux doesn't seem to have any kind of tests. How can they afford not to have regression tests for bugs they fixed? How do they know that this bug fix didn't break anything? What does "never break userspace" even mean if there is no way to check whether userspace has been broken?

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#44

Post by uep »

pjc50 wrote: post #41
erikb wrote: post #31

Earlier quoted context omitted.

Just to give a picture how similar situations play out in closed source environments: They would have a key account manager who's live they would turn into hell. And depending on the size of gitlab's business in relation to the customers of the software provider it would have more or less effect towards software development. Reaction might range from having a fix in 2 days plus on-site engineer/consultant visits plus…

This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it. (I got as far as disassembling their DLLs to point to the exact problem)

Yeah, I don't really buy the parent comment's insinuation that Microsoft GitHub would be more likely to fix it than GitHub previously. My company has shipped hundreds of thousands of devices with a Microsoft OS on it, and they virtually never fixed anything reported. At least one bug was very serious, and we thought they would have to fix it, but it just didn't happen.

What's hundreds of thousands to the hundreds of millions they ship every year?

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#45

Post by drewg123 »

kelleyk wrote: post #4
drewg123 wrote: post #2

I spend my days chasing bugs like this in the FreeBSD kernel, and make heavy use of dtrace. I expect that using something like bpftrace(1) might have accelerated their debugging as compared to inserting stack traces and prints... (1): http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux...

Out of curiosity, what do you do that you get to do that? Sounds fascinating---I've had lots of fun with dtrace (and, more recently, bcc).

I work for Netflix on Open Connect, the Netflix CDN. We run a modified FreeBSD on our CDN nodes. My job involves improving performance and scalability in the FreeBSD kernel. See https://medium.com/netflix-techblog/serving-100-gbps-from-an... Very much like the NFS article author describes, I also tend to find "naive" solutions that end up being polished a lot before landing upstream.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#46

Post by chronolitus »

virtualized wrote: post #43

I always wonder why Linux doesn't seem to have any kind of tests. How can they afford not to have regression tests for bugs they fixed? How do they know that this bug fix didn't break anything? What does "never break userspace" even mean if there is no way to check whether userspace has been broken?

I guess the user is the test.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#47

Post by AceJohnny2 »

erikb wrote: post #31
atq2119 wrote: post #26

It's amazing how much we take the power of free and open source software for granted these days. Imagine this same scenario if GitLab was using a closed source operating system. Would they have been able to track this down? Quite unlikely, but maybe if they were even more persistent and got lucky. Would they have been able to fix it? Absolutely not. They'd be at the mercy of the vendor.

Just to give a picture how similar situations play out in closed source environments: They would have a key account manager who's live they would turn into hell. And depending on the size of gitlab's business in relation to the customers of the software provider it would have more or less effect towards software development. Reaction might range from having a fix in 2 days plus on-site engineer/consultant visits plus…

> but open source certainly helps balancing out the odds for different competitors.

Assuming all competitors have the same engineers who are competent in the various technologies involved in debugging this (at a glance: filesystem operations, strace, wireshark, linux kernel compilation and modules, Google Cloud Platform...), know how to contact and approach the open-source maintainers.

... and most importantly: have the time to dedicate to such a debugging task.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#48

Post by specialist »

jlokier wrote: post #39

Really nice. I have more respect for GitLab now. That's a great write-up, and it led me to read some of their other nice reports too. It's not exactly new for NFS to have cache coherency "surprises". But it should have "close-to-open" coherency at least, and the bug found by GitLab fails even that. Here's an anecdote. A Mac client talking to Samba on Linux. The client deletes random files that the client isn't even l…

Great war story. Agree with need for access to source.

--

During the early Java WORA culture wars, Bill Joy's wisdom about NFS has always stuck with me:

Interoperability is hard.

Despite having access to source code, a stable spec, working reference implementations, testing suites, and aggressive evangelism, getting everyone's NFS implementations to interoperate was a major challenge.

https://en.wikipedia.org/wiki/Network_File_System

--

I continue to think the authoritative history of NFS would become a seminal text book. A useful guide for the younguns about to embark on grand new world changing adventures. Many, many other protocols (DNS, TCP, HL7, CORBA...) have faced the same challenges. But my hunch is NFS is a superset, hitting every pain point.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#49

Post by peterwwillis »

erikb wrote: post #31
atq2119 wrote: post #26

It's amazing how much we take the power of free and open source software for granted these days. Imagine this same scenario if GitLab was using a closed source operating system. Would they have been able to track this down? Quite unlikely, but maybe if they were even more persistent and got lucky. Would they have been able to fix it? Absolutely not. They'd be at the mercy of the vendor.

Just to give a picture how similar situations play out in closed source environments: They would have a key account manager who's live they would turn into hell. And depending on the size of gitlab's business in relation to the customers of the software provider it would have more or less effect towards software development. Reaction might range from having a fix in 2 days plus on-site engineer/consultant visits plus…

Or another example in closed source:

We had the source code for an API that hooked into a proprietary library. We found a bug in the library. I don't think we had a support contract, and the issue was affecting production. Fixing it could have entailed some decompiling of the library, identifying the bad function, writing a workaround, and shoving it all into a new library. But I didn't have the expertise for all that, so instead I hacked up the API with a different workaround, essentially killing off some functionality, which avoided the bug. The application worked again, and we went on with life.

Another example: a proprietary extension to a tool did data replication. Under certain circumstances, data replication would fail, and the loss of data meant we would have to full-sync all data, taking up to four days. We reported the bug to the company. They determined it was a "minor error" and said the fix would arrive in the next release, in six months. So we identified a workaround (add cacheing, monitor for potential service disruption, restart services to re-connect networks before cache would empty) and implemented it until the fix could be delivered.

Regardless of who fixes the bug or how, the amount of time and money you invest in the fix matters. If a workaround saves you time and money by deferring the cost of the fix, that's often an acceptable solution. In this case, if the issue was affecting customers in production, blocking 'git gc' just for affected customers may have been a perfectly good workaround while whoever owned the NFS Client code figured out and implemented a fix.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#50

Post by lbotos »

avar wrote: post #38

A tangential question, the post links to an earlier post[1] saying that GitLab itself doesn't use NFS anymore, pointing out that they migrated to Gitaly. But ultimately Gitaly will need to do a local FS operation, so there's still the problem of ensuring HA for a given repository. GitHub solved this by writing their own replication layer on top of Git[2], but what's GitLab doing? Manually sharding repos on local FS's…

We are working on Gitaly HA. You can check out the Epic here:

https://gitlab.com/groups/gitlab-org/-/epics/289

Post reply on HN