Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent two weeks hunting an NFS bug in the Linux kernel

about.gitlab.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#61

Post by alexeiz »

Git on NFS has proved to be a full of annoyances. Though I haven't seen any Git repo integrity issues on NFS, normal Git operations can be so slow on NFS it's infuriating (especially when the size of your repository is sufficiently large). Do you like that fancy Bash prompt showing 'git status' for the repository you currently work with? Forget about it, if you're on NFS. Or get used to waiting a couple of seconds after each Bash command while it's blocked on 'git status'. The solution is just to avoid NFS altogether and work with Git repos on a local filesystem.

To add insult to injury, this is an example of how people like to work with Git repositories at our company:

* Clone a Git repo into $HOME to work with it on different Linux hosts. $HOME is an NFS automount so that you have the same home environment on any host you log in to.

* $HOME is also exposed to Windows desktop machines via SMB. So convenient, right? You can edit source code in your favorite Windows IDE now!

Imagine their surprise when they make yet another Git commit with garbage in it. CR/LF, file mode bits are all messed up. Sometimes a file change on Windows take a long time to propagate to NFS, or worse yet there can be some garbage at the end of the file. Combine this with a common practice of committing with 'git commit -am' without even looking at the diff and you get a recipe for disaster.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#62

Post by idclip »

pjc50 wrote: post #54
idclip wrote: post #34

Earlier quoted context omitted.

i dont believe there is sense in naming it so people can scape goat it anf solve things by avoiding one bad university. because i think the BA/MA system is a road paved with good intentions. but its leading us to hell. naming any singular entity would just make us think its them to blame, and i believe the problem is endemic. i sat once on a table with PHD students and complained about the quality of introductory cou…

> "a university does not prepare for work! it prepares for research!" This is what universities have always believed that they are for. PhD courses in particular. There used to be a separate category of school that was both technical and employment focused; in the UK these were called "polytechnics", in the US they would be things like the Texas Agricultural and Mining College. For complex reasons due to both the cla…

thats actually interesting!

i have a feeling they will be making a comeback woth a vengeance, but maybe not in our time.

today i just do what i can when a youth comes to me for advice.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#63

Post by tinus_hn »

pjc50 wrote: post #41
erikb wrote: post #31

Earlier quoted context omitted.

Just to give a picture how similar situations play out in closed source environments: They would have a key account manager who's live they would turn into hell. And depending on the size of gitlab's business in relation to the customers of the software provider it would have more or less effect towards software development. Reaction might range from having a fix in 2 days plus on-site engineer/consultant visits plus…

This very much depends on how big you are and how much you're willing to spend. At a medium-size all-Microsoft shop with 100+ MSDN licensees, I found a bug in WINCE7's handling of "structured exceptions", and eventually got them to acknowledge it but they never fixed it. (I got as far as disassembling their DLLs to point to the exact problem)

Supposedly if you buy support incidents they should do something. I don’t know if that is actually true in practice though.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#64

Post by tinus_hn »

batbomb wrote: post #21

I had been experiencing a similar bug that we reported to Red Hat after being stumped. It started occurring out of the middle of nowhere but it would happen only in .01% of our jobs we launched into a batch farm. We launch about 15k batch jobs a day, and it was enough to be a problem. Before a job was launched, a daemon pre-staged some job contents (logfiles, env, etc..) and started writing out to a job summary file.…

Too bad that kind of thing is difficult to catch using static analysis.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#65

Post by tinus_hn »

raverbashing wrote: post #15

To be honest, NFS is usually more pain than it is worth it. (But hey, at least it's not iSCSI) Yes please let the default not allow me to unmount a fs from a server that died.

It is very good at doing exactly what you don’t want though.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#66

Post by tracker1 »

alexeiz wrote: post #61

Git on NFS has proved to be a full of annoyances. Though I haven't seen any Git repo integrity issues on NFS, normal Git operations can be so slow on NFS it's infuriating (especially when the size of your repository is sufficiently large). Do you like that fancy Bash prompt showing 'git status' for the repository you currently work with? Forget about it, if you're on NFS. Or get used to waiting a couple of seconds af…

I wish that git for windows would just change the default line endings to \n. Most modern editors should also make this shift as a default, or if there's no \r\n combination in an existing doc, just use \n.

I, generally set this as default, and sometimes forget on a new machine... I tend to prefer those tools/programs that work in Windows, Mac and Linux even if not quite as good, so concerns about where I am is less. I use windows keyboard on mac, and change the mapping... only gotcha is when I need ^C in a terminal on mac, the muscle memory screws me up sometimes switching from working at home (mac or linux) to working at work (windows).

Some quirkiness with git's bash on windows (my shell default) get me sometimes too.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#67

Post by justinclift »

stanhu wrote: post #53
justinclift wrote: post #9

The Red Hat BZ is "Access Denied", so can't see if it's fixed in RHEL & CentOS yet: https://bugzilla.redhat.com/show_bug.cgi?id=1648482 :(

You can sign up for an account to see the bug report status. The patch has not yet been backported.

I have an account. Why would that make any difference?

RH BZ's are (or used to be) public by default, unless they're manually changed. eg for security related things

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#68

Post by romeisendcoming »

jabl wrote: post #59
romeisendcoming wrote: post #58

Earlier quoted context omitted.

Well enough isn't good enough for production. Having worked in high volume, highly available environments for years soho scenarios are not good examples. Real world issues with complex NFS environments (mixed nfs3/4 + krb5p and multiple OS'es + automounters) or pNFS and gluster require more than tuning mount options. Tuning NFS for a latency averse and throughput intensive application operating on large netcdf and hd…

> Well enough isn't good enough for production. How informative. > Having worked in high volume, highly available environments for years soho scenarios are not good examples. FWIW, I wasn't talking about SOHO. At least in my experience, defaults work well for home & shared work dirs for O(10k) users (not all simultaneously active, though). HA is a pain, though, if you want to DIY, I'll grant you that. > Real world is…

* routine buffer tweak is soho speak. * defaults don't work esp in mixed nfs 3/4 on linux across mixed 1/10 gb segment subnet boundaries. Try and get back to me. You will DOS your file service. * krb5p standard? First I've heard of it. FreeBSD won't do krb5p at nfs4 vanilla via linux nfs server. * Would never do it again. gluster is a shit storm of problems but nice when it works.

Re: How we spent two weeks hunting an NFS bug in the Linux kernel

#69

Post by romeisendcoming »

jabl wrote: post #40
romeisendcoming wrote: post #35

Earlier quoted context omitted.

Totally disagree. You have to understand how NFS caching effects your application and what kind of performance and security/availability you need. At that point you can make NFS work for you. You also have to make good decisions about how you implement. This type of use case (long open file waits with concurrent access) are a nightmare with any shared filesystem. Most approaches dealing with possibly stale content ar…

> This type of use case (long open file waits with concurrent access) are a nightmare with any shared filesystem. Most approaches dealing with possibly stale content are shoulder shrugs. Well, it is possible to handle correctly, e.g. Lustre. Lustre, however, is very complex compared to NFS, so there's absolutely a price to be paid. NFS implements close-to-open consistency, which is much weaker than full cache coheren…

You want a network shared filesystem then design your data for it.
Post reply on HN