GitLab has a strong engineering team. I appreciate this article. For those with experience, what's the best approach to introducing a documentation / "writing up a post-mortem culture" into a company that traditionally doesn't value these things?
How we spent two weeks hunting an NFS bug in the Linux kernel
11–20 of 69 posts
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#12Great post! In my own experience of working with NFS version 4 servers, we discovered several bugs that have been actually fixed in latest version of kernels. The unfortunate thing is that most enterprises still run old CentOS / Redhat release kernels that although are stable, but yet lack several of these fixes.
I don't have a lot of experience with NFS aside from a few machines that don't see insane use, but it's surprising to me how v4 implementations seem to introduce such instability. I had an experience a few years ago with a Mac client, quitting vim would cause a kernel panic. NFS v3 did fine.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#13GitLab has a strong engineering team. I appreciate this article. For those with experience, what's the best approach to introducing a documentation / "writing up a post-mortem culture" into a company that traditionally doesn't value these things?
Start doing it. Celebrate it. Reward it. Keep doing it.
Like all culture changes it isn't complicated, just hard.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#14I love these posts ! Once or twice I was lucky enough to have had a somewhat similar intense debug-puzzle's to solve myself. Wonderful times :)
strace is a god send !
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#15To be honest, NFS is usually more pain than it is worth it. (But hey, at least it's not iSCSI)
Yes please let the default not allow me to unmount a fs from a server that died.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#16Representing such an example of work in a job application / resume / interview would be more valuable to me than a college degree. Due diligence and persistence — in the face of real-world, difficult, hundreds-of-moving-parts technical issues — are worth every penny.
EDIT: Yes, college degrees require due diligence and persistence, but they offer no indication of the willingness to exercise those skills _after_ college. This work does.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#17The Red Hat BZ is "Access Denied", so can't see if it's fixed in RHEL & CentOS yet: https://bugzilla.redhat.com/show_bug.cgi?id=1648482 :(
It’s still in NEW - don’t see any chatter yet
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#18To be honest, NFS is usually more pain than it is worth it. (But hey, at least it's not iSCSI) Yes please let the default not allow me to unmount a fs from a server that died.
Does anyone know why NFS is such a pain? In the past (10 years ago) I just assumed I was doing it wrong and stopped using it, and have not used it since.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#19I spend my days chasing bugs like this in the FreeBSD kernel, and make heavy use of dtrace. I expect that using something like bpftrace(1) might have accelerated their debugging as compared to inserting stack traces and prints... (1): http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux...
We're getting there! :) I've asked my team to balance learning K8s with new learning lower level debugging tools like bcc and it's cohort.
Re: How we spent two weeks hunting an NFS bug in the Linux kernel
#20To be honest, NFS is usually more pain than it is worth it. (But hey, at least it's not iSCSI) Yes please let the default not allow me to unmount a fs from a server that died.
Does anyone know why NFS is such a pain? In the past (10 years ago) I just assumed I was doing it wrong and stopped using it, and have not used it since.
Posix semantics and networks just dont play well together. Notice that binary storage systems these days avoid the filesystem integration (s3, etc.)