Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why doesn't `kill -9` always work?

noah.org

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Why doesn't `kill -9` always work?

#41

Post by yebyen »

SEMW wrote: post #38
laumars wrote: post #35

Earlier quoted context omitted.

> Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error. Any storage medium is subject to fail, the issue isn't specific to remote file systems. In fact I personally consider the Plan 9 methodology towards file systems to be one of the greatest ideas it's imparted on Linux and Unix. So I'm all in favour of the local file systems, services and remote objects and file syste…

> People often cite the speed of light when talking about electronics when actually electrons don't travel at the speed of light (they have mass) Actual electrons move pretty slowly - of the order of millimetres per hour. That speed has nothing to do with the speed the signal propagates down the wire. The speed the signal is propagated is the speed the electromagnetic wavefront moves along it. Which is mostly limited…

Damn, you beat me to it. I think you did a better job explaining than I would have, too. An upvote for you, sir!

For any skeptics or laymen who might have been lost at 'electromagnetic wavefront', or who are wondering why electrons would seem to move so slowly when we think of electricity (and even free electrons) as moving so fast, the analogy of the broomstick really does not do it justice, since you can't see the broomstick oscillating very quickly as you might have guessed the electrons must really do to satisfy Heisenberg:

http://en.wikipedia.org/wiki/Particle_in_a_box

http://www.physicsforums.com/showthread.php?t=121456

The keyword is "electron drift velocity"

The important concept to understand is that information does not move trapped inside of individual electrons, and thus is not at all limited by electron drift velocity.

Re: Why doesn't `kill -9` always work?

#42

Post by SummonJetTruck »

SEMW wrote: post #38
laumars wrote: post #35

Earlier quoted context omitted.

> Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error. Any storage medium is subject to fail, the issue isn't specific to remote file systems. In fact I personally consider the Plan 9 methodology towards file systems to be one of the greatest ideas it's imparted on Linux and Unix. So I'm all in favour of the local file systems, services and remote objects and file syste…

> People often cite the speed of light when talking about electronics when actually electrons don't travel at the speed of light (they have mass) Actual electrons move pretty slowly - of the order of millimetres per hour. That speed has nothing to do with the speed the signal propagates down the wire. The speed the signal is propagated is the speed the electromagnetic wavefront moves along it. Which is mostly limited…

With a broomstick, wouldn't it be the speed of sound, since it's a pressure wave?

Re: Why doesn't `kill -9` always work?

#43

Post by anonymous »

dfox wrote: post #18
ambrop7 wrote: post #14

I consider any uninterruptable sleep in the kernel a bug. There's no technical reason a process waiting for a resource (e.g. disk I/O) couldn't be killed on the spot, leaving the resource on its own. If it can't be, it just means it hasn't been implemented in the kernel.

It's bug motivated by compatibility. On original 70's implementations of Unix, file system I/O mostly led to busy wait in kernel and thus was not interruptible because it was simply not possible and there were applications that relied on this behavior. On UNIX, signal received during system call generally causes the kernel to abort whatever it was doing and requires application to deal with that situation and restart…

So having a system that cannot be safely shut down because someone has opened a file handle on a remote fs that became unavailable is preferable to special-casing one signal?

But by all means, if the signal thing bothers you - fine, I just really want some way to stop the process. Let's add two new system calls, something like "terminate with extreme prejudice" (abbr. twep) and "unmount without mercy" (abbr. uwm).

Re: Why doesn't `kill -9` always work?

#44

Post by SEMW »

SummonJetTruck wrote: post #42
SEMW wrote: post #38

Earlier quoted context omitted.

> People often cite the speed of light when talking about electronics when actually electrons don't travel at the speed of light (they have mass) Actual electrons move pretty slowly - of the order of millimetres per hour. That speed has nothing to do with the speed the signal propagates down the wire. The speed the signal is propagated is the speed the electromagnetic wavefront moves along it. Which is mostly limited…

With a broomstick, wouldn't it be the speed of sound, since it's a pressure wave?

Fair point. For the analogy I was going for a hypothetical rigid broomstick, made of stuff that's as incompressible as physically possible. The speed of sound in that would be c. Yeah, that isn't very realistic, but I don't think that hurts the point it was illustrating.

Re: Why doesn't `kill -9` always work?

#45

Post by agwa »

syncsynchalt wrote: post #25
agwa wrote: post #12

Earlier quoted context omitted.

I don't understand it either. The only syscalls going on there are reads from /dev/random and writes to /dev/null - both of those can be interrupted (in fact writes to /dev/null should be instantaneous). I think the author may be conflating applications blocked in system calls (since reads from /dev/random will block if the system lacks entropy) with applications blocked in uninterruptible system calls.

There are no reads from /dev/random successfully happening in the example (not after the first few blocks, anyway). /dev/random reads from an entropy pool that is quickly exhausted and slowly filled. The kernel will lock a process in D state while waiting for entropy. If you're expecting a stream of pseudo-random data then you can get that by directing /dev/ urandom to /dev/null.

> The kernel will lock a process in D state while waiting for entropy.

No it doesn't, at least not in Linux 3.0.57 or any other kernel I can remember for the last many years. It blocks if it needs entropy, but it's interruptible meaning it's in the S state, not the D state, and can be killed.

Re: Why doesn't `kill -9` always work?

#46

Post by agwa »

asveikau wrote: post #27
dfox wrote: post #18

Earlier quoted context omitted.

It's bug motivated by compatibility. On original 70's implementations of Unix, file system I/O mostly led to busy wait in kernel and thus was not interruptible because it was simply not possible and there were applications that relied on this behavior. On UNIX, signal received during system call generally causes the kernel to abort whatever it was doing and requires application to deal with that situation and restart…

> implementations of stdio in libc generally do the right thing Tell me, what is the "right thing" for stdio to do when it sees EINTR? It strikes me that this can't really be solved at the library level. There are times when you'll want to retry and there are times when you'll want to drop your work and surface the error to the caller. Doesn't seem to me like a library can decide which is which. Which is probably why…

The right thing is almost always to retry the syscall. Syscalls on Unix return EINTR because it makes the kernel simpler, which was a key design goal in Unix[1]. If you need to do something when a signal fires, you do it in a signal handler (relying on EINTR instead of a signal handler is error-prone because if the signal fires between syscalls you lose it).

That's the theory anyways - in practice it's really hard to use signal handlers to do stuff because of things like threads and async-signal safety. There are newer syscalls like pselect() which let you atomically unblock signals, execute the syscall, and re-block the signals, meaning EINTR can be used reliably, and then there's the even newer Linux-only signalfd() syscall which lets you receive signals via a file descriptor. But stdio is still very much oriented for the older signal handler approach.

[1] See the paper "The Rise of ``Worse is Better''" http://www.stanford.edu/class/cs240/readings/worse-is-better...

Re: Why doesn't `kill -9` always work?

#47

Post by dredmorbius »

laumars wrote: post #35
dredmorbius wrote: post #21

Earlier quoted context omitted.

Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error. Time, and speed of light, ultimately matter. If you need assurance, find a way of getting reliability in your system through redundancy and locality. Distinguish between "task has been delegated" and "task has been confirmed completed". Down any other path runs pain, and anyone who tells you otherwise is selling somet…

> Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error. Any storage medium is subject to fail, the issue isn't specific to remote file systems. In fact I personally consider the Plan 9 methodology towards file systems to be one of the greatest ideas it's imparted on Linux and Unix. So I'm all in favour of the local file systems, services and remote objects and file syste…

My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues.

Speed of light matters in that it sets an lower absolute limit on the latency of connections between two points, particularly if they're outside of a single datacenter. While non-local storage can be faster (and more reliable) than local storage within a well-architected datacenter environment, even a link of a few tens of miles, let alone thousands, will introduce SOL latencies, carrier inconsistencies, and switching and other equipment idiosyncrasies. Your point that actual propagation rarely hits SOL speeds only compounds the problem.

I've looked at iSCSI. NFS, for its warts, allows multiple concurrent access, iSCSI binds storage to a single system. Management/diagnosis tools, vendor sophistication, documentation, and support are sorely lacking (or were on my last exposure). NFS may suck, but we can quantify, analyze, diagnose, and understand its suckage.

My own preference is for distributed network/application based storage. Caching for repeated static queries. Direct remote database access (if your network and remote storage are fast enough for NFS, they're fast enough to put the DB server directly on hot storage), Hadoop/MapReduce type solutions for higher-speed access, git or other change-and-push state for less aggressive changes. Getting to my earlier point: this requires architecting your application to take account of remote storage capabilities and limitations. It need not be specific to a given technology (and ideally you'd abstract out to the level of "a cache", "a database", "a keypair storage", "a versioning system"), but you're also not just expecting local filesystem semantics and reliability.

Ironically, gitfs is a thing.

Re: Why doesn't `kill -9` always work?

#48

Post by asveikau »

agwa wrote: post #46
asveikau wrote: post #27

Earlier quoted context omitted.

> implementations of stdio in libc generally do the right thing Tell me, what is the "right thing" for stdio to do when it sees EINTR? It strikes me that this can't really be solved at the library level. There are times when you'll want to retry and there are times when you'll want to drop your work and surface the error to the caller. Doesn't seem to me like a library can decide which is which. Which is probably why…

The right thing is almost always to retry the syscall. Syscalls on Unix return EINTR because it makes the kernel simpler, which was a key design goal in Unix[1]. If you need to do something when a signal fires, you do it in a signal handler (relying on EINTR instead of a signal handler is error-prone because if the signal fires between syscalls you lose it). That's the theory anyways - in practice it's really hard to…

Yes, I'm aware, "almost always". Not always, though. I was thinking specifically of an application that might want to use signals to cancel blocking I/O and continue running.

(PS: When I wrote my reply I was also already familiar with your linked article, the challenges of signal safety, and the signalfd() syscall. Surely an interesting set of topics but I still maintain that a library doesn't really have a "good" way to deal with EINTR, especially if all it does is wrap read or write.)

Re: Why doesn't `kill -9` always work?

#49

Post by laumars »

dredmorbius wrote: post #47
laumars wrote: post #35

Earlier quoted context omitted.

> Expecting NFS (or any other remote filesystem) to behave as if it were local is a fundamental error. Any storage medium is subject to fail, the issue isn't specific to remote file systems. In fact I personally consider the Plan 9 methodology towards file systems to be one of the greatest ideas it's imparted on Linux and Unix. So I'm all in favour of the local file systems, services and remote objects and file syste…

My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues. Speed of light matters in that it sets an lower absolute limit on the latency of connections between two points, particularly if they're outside of a single datacenter. While non-local storage can be faster (and more reliable) than local storage within a well-architected datacenter env…

    > My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues.
Usually that's the fault of lower down in the chain than with NFS itself. If you're building a home brew SAN using NFS, then there's a lot of kernel tweaks and such like that you can implement. The Linux kernel is a great all-rounder, but if you're building your own enterprise-grade appliances then there's a lot more work required than just vi'ing /etc/exports.

But anyone using NFS to run databases or for caches needs their head examining. It's purely a protocol for file system sharing, caching should be ramdisk, databases should be using dedicated TCP/IP channels preferred by the RMDBS in question and so on. Using NFS as a blanket solution like that is a little like running websites over NFS and wondering why PHP isn't compiling properly. In fact some versioning systems do use HTTP as their method for syncing.

Re: Why doesn't `kill -9` always work?

#50

Post by dredmorbius »

laumars wrote: post #49
dredmorbius wrote: post #47

Earlier quoted context omitted.

My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues. Speed of light matters in that it sets an lower absolute limit on the latency of connections between two points, particularly if they're outside of a single datacenter. While non-local storage can be faster (and more reliable) than local storage within a well-architected datacenter env…

> My experience with NFS failures is that they're an order or two of magnitude higher (and much more binary) than local storage issues. Usually that's the fault of lower down in the chain than with NFS itself. If you're building a home brew SAN using NFS, then there's a lot of kernel tweaks and such like that you can implement. The Linux kernel is a great all-rounder, but if you're building your own enterprise-grade…

I'm referring to an enterprise implementation comprising over 5000 nodes and somewhere in the neighborhood of 300-400 NFS mounts under typical use (~60-70 at boot, additional mounts via autofs under typical workloads).

Issues: ad-hocery, former Sun shop, growth-through-acquisition (including acquiring the application infrastructure of said firms), 20 years of legacy, much staff attrition (no more than normal, but even 5-10% means virtually complete turnover within that period), geographically distributed network (3-4 continents), etc.

Pretty much a worst pathological case, except that it's very standard in many, many established firms.

Post reply on HN