Why doesn't `kill -9` always work? • Hacker News Board

Re: Why doesn't `kill -9` always work?

#51

Post by dredmorbius » Mon, Jan 21, 2013, 11:37 PM UTC

smutticus wrote: post #32

dredmorbius wrote: post #21

Earlier quoted context omitted.

Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error. Time, and speed of light, ultimately matter. If you need assurance, find a way of getting reliability in your system through redundancy and locality. Distinguish between "task has been delegated" and "task has been confirmed completed". Down any other path runs pain, and anyone who tells you otherwise is selling somet…

In my experience NFS is fine as long as the traffic does not need to traverse a WAN link. The problem, as I see it, is that once something is available via NFS everyone wants to mount it from everywhere. NFS simply does not deal well with high RTT connections.

Good luck with that assumption.

/net

Re: Why doesn't `kill -9` always work?

#52

Post by laumars » Tue, Jan 22, 2013, 7:07 AM UTC

dredmorbius wrote: post #50

laumars wrote: post #49

Earlier quoted context omitted.

> My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues. Usually that's the fault of lower down in the chain than with NFS itself. If you're building a home brew SAN using NFS, then there's a lot of kernel tweaks and such like that you can implement. The Linux kernel is a great all-rounder, but if you're building your own enterprise-grade…

I'm referring to an enterprise implementation comprising over 5000 nodes and somewhere in the neighborhood of 300-400 NFS mounts under typical use (~60-70 at boot, additional mounts via autofs under typical workloads). Issues: ad-hocery, former Sun shop, growth-through-acquisition (including acquiring the application infrastructure of said firms), 20 years of legacy, much staff attrition (no more than normal, but eve…

I see.

Will some kind of distributed file system work here? If you can clustered your NFS server you'd (hopefully) end up with more localised nodes for your client to connect to (a little bit like how a CDN works). For what it's worth, I wouldn't even run 5000 simultaneous HTTP requests on a single node in any of my web farms, let alone on a single NFS server.

Also have you looked into kernel level tweaks? I'm guessing you're either running Solaris (being a former Sun shop) and while I am a sysadmin for Solaris, I've not needed to get this low level before, but certainly on Linux, there's a lot of optimisations that can be made to the TCP/IP stack that would improve the performance in this specific scenario.

I do agree with you that 400->5000 active NFS connections does push the realm of practical use, but I don't think that dismisses NFS entirely; it still outperforms all other network mounted file systems.

Re: Why doesn't `kill -9` always work?

#53

Post by dfox » Tue, Jan 22, 2013, 4:26 PM UTC

asveikau wrote: post #48

agwa wrote: post #46

Earlier quoted context omitted.

The right thing is almost always to retry the syscall. Syscalls on Unix return EINTR because it makes the kernel simpler, which was a key design goal in Unix[1]. If you need to do something when a signal fires, you do it in a signal handler (relying on EINTR instead of a signal handler is error-prone because if the signal fires between syscalls you lose it). That's the theory anyways - in practice it's really hard to…

Yes, I'm aware, "almost always". Not always, though. I was thinking specifically of an application that might want to use signals to cancel blocking I/O and continue running. (PS: When I wrote my reply I was also already familiar with your linked article, the challenges of signal safety, and the signalfd() syscall. Surely an interesting set of topics but I still maintain that a library doesn't really have a "good" wa…

Simplest solution would be for libc to export some flag that could be set in signal handler signifying that I/O operation should be aborted.

as for the simpler kernel, I think that windows NT/VMS solution where user code has to explicitly block on I/O completion is simpler kernel-wise, but leads to unnecessary complexity in applications (which is abstracted away by winapi, but it's sometimes leaky abstraction). On the other hand, most common application for interrupting syscalls is timeouts and then killing the thread is most often what you want.

In all, EINTR is not way to find out that there was an signal during syscall but an hack to get process to meaningful state the easiest possible way when signal handler runs. By the way for some syscalls post-2.6 linux does something reasonably similar to ITS' pclusering transparently without returning EINTR.