Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why doesn't `kill -9` always work?

noah.org

Post reply on HN
Print view View original

1–10 of 53 posts

Re: Why doesn't `kill -9` always work?

#2

Post by a_bonobo »

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

Re: Why doesn't `kill -9` always work?

#3

Post by jsnell »

a_bonobo wrote: post #2

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

So that whole section doesn't make a lot of sense to me. Running strace on an unkillable process tends to produce no output (if the process was actually making new system calls, it wouldn't be wedged). And worse, my experience is that attaching to a wedged process with ptrace() usually does nothing at all except also hang the attaching the process. This also applies to gdb.

And finally, even if attaching worked, getting a disassembly of the current PC (in this case the syscall trampoline) would tell nothing useful about what's going on.

Re: Why doesn't `kill -9` always work?

#4

Post by nonane »

a_bonobo wrote: post #2

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

It think one of the reasons the kernel can't kill a process is because one of the process's threads is blocked inside a kernel call (not completely sure about this). Hes using ps's 'wchan' option to get the address of the kernel function that the process is currently blocked or sleeping on. After he gets the wchan address, he uses gdb to map the address to function name.

Taken from: http://unixhelp.ed.ac.uk/CGI/man-cgi?ps

nwchan WCHAN address of the kernel function where the process is sleeping (use wchan if you want the kernel function name). Running tasks will display a dash ('-') in this column.

Re: Why doesn't `kill -9` always work?

#6

Post by mbell »

a_bonobo wrote: post #2

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

I believe the author is using that as an example of a command that will 'wedge' if you tried to kill -9 it.

The rest of it is an example of how to dump the cause of a process that is wedged.

"/proc/{pid}/wchan" contains the name of the current syscall a process is executing.

Basically you should be able to use everything but the first line of that section as a shell script(with some modifications) to determine the cause of a wedged process.

Re: Why doesn't `kill -9` always work?

#7

Post by dllthomas »

a_bonobo wrote: post #2

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

Not sure just how much you do/don't understand, or how much others will/won't understand, so I'll run through line by line:

    PID=$!
grabs the PID of the process you just spawned (in the background, with &) into shell variable PID

    CMDLINE="!-2"
grabs the full line you just ran (before the line storing PID) with shell history expansion

    CMD=${CMDLINE%% *}
expands the CMDLINE variable, replacing everything after the first space (so CMD now has "cat") with bash trickery

    WCHAN=$(cat /proc/${PID}/wchan)
grabs the name of the currently executing syscall for the process (at least, according to http://www.lindevdoc.org/wiki/proc/pid/wchan)

    echo "command: ${CMD}, pid: ${PID}, wchan: ${WCHAN}"
prints the info we've grabbed

    strace -p ${PID}
connects a trace to the process to see what it's doing

    gdb ${CMD} ${PID}
connects to the process (gdb needs program name and can be given a pid to connect to)

    (gdb) disassemble
prints the actual (assembler) code being run. In this case, I think all we get from the output is that it's in fact in the middle of some syscall - you'd have to check registers and syscall tables to determine which.

As others have mentioned, much of this is less useful than implied in the face of an actual wedged process.

Tangentially, using gdb to attach to running processes is a very powerful technique - I've been able to get line numbers out of running bash scripts.

Re: Why doesn't `kill -9` always work?

#8

Post by agwa »

Do not mount NFS with "soft" unless you really know what you're doing. NFS' behavior to hang is not "stupidity" - it's actually one of the best things about NFS. Applications do not deal well with failed reads/writes. If there's a brief network interruption or the NFS server goes down, it's WAY safer to cause applications to hang until the server comes back. Since NFS is a stateless protocol, when the server comes back, I/O resumes as if nothing ever happened. This helps make using NFS feel more like using a local filesystem. Otherwise it becomes a very leaky abstraction.

Not being able to kill processes stuck on NFS I/O is annoying though, so you can mount with the "intr" option and that makes such processes killable. However, since Linux 2.6.25, you don't even need this and SIGKILL can always kill applications stuck in NFS I/O.

Re: Why doesn't `kill -9` always work?

#9

Post by mbell »

nonane wrote: post #4
a_bonobo wrote: post #2

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

It think one of the reasons the kernel can't kill a process is because one of the process's threads is blocked inside a kernel call (not completely sure about this). Hes using ps's 'wchan' option to get the address of the kernel function that the process is currently blocked or sleeping on. After he gets the wchan address, he uses gdb to map the address to function name. Taken from: http://unixhelp.ed.ac.uk/CGI/man-c…

/proc/{pid}/wchan contains the function name, not its address.

In this example he's just printing the syscall that his example process is running.

The calls to strace and gdb are disjoint, he's cut out a lot of stuff here, probably why its confusing. In the commands he's used strace would dump all the syscall details to the console and GDB would just attach to the process and enter its normal console (presumably hes determined hes stuck in a loop so hes just using disassemble).

Re: Why doesn't `kill -9` always work?

#10

Post by a_bonobo »

dllthomas wrote: post #7
a_bonobo wrote: post #2

Can anyone explain the "Why is a process wedged?" part? I do understand that piping from /dev/random to /dev/null is going to run forever, but I do not understand the gdb-output, nor what that has to do with the rest of the text.

Not sure just how much you do/don't understand, or how much others will/won't understand, so I'll run through line by line: PID=$! grabs the PID of the process you just spawned (in the background, with &) into shell variable PID CMDLINE="!-2" grabs the full line you just ran (before the line storing PID) with shell history expansion CMD=${CMDLINE%% *} expands the CMDLINE variable, replacing everything after the first…

Thank you for that! Makes it clear, I guess I have to properly learn gdb's output... I work in bioinformatics, we never go that low.
Post reply on HN