Live data from Hacker News

When should I not kill -9 a process?

unix.stackexchange.com

31–40 of 85 posts

Re: When should I not kill -9 a process?

#31
post #10
post #4

As if often the case with stackoverflow answers, all of them are wrong in different ways. You should only kill -9 when every other signal the program is likely to respond to has not worked. kill -9 is likely to leave program in a state that requires manual intervention, especially if that program is a database. If you're a developer, before you kill -9 a program send SIGTERM (ie kill without args or kill -15). If the…

You can't really corrupt a database that easily, can you? That's half the point of using a database, so you have transactions, etc.

I've never corrupted the DB, but I have ended up with inconsistencies because I needed to do several related queries in a row. Because I didn't send them as a single transaction from my application some of them were run, some not.

Edit: Especially when you have multiple systems that should be consistent. Deleting a user from the DB, but being killed before it can be removed from our auth system, for instance.

Re: When should I not kill -9 a process?

#32
post #15
post #7

Over 15 years ago, as a teenager, I taught Linux / UNIX Admin courses, and worked as a consultant advising folks, and in the late 90s I was very adamant that you should never -9 anything unless you know exactly what you are doing. As infrastructures have grown, and I have managed large applications involving tens to hundreds, often over a thousand servers, and I have grown to accept that a power supply can fail and a…

kill -9 should never be worse than pulling the power plug No, it shouldn't, but just pulling the power plug isn't exactly recommended behaviour, either. There aren't many admins out there who will happily yank the power cord out of their desktops when they want to power it down. Last night I spent several hours getting a server back into gear after a 'pulled power plug' event. A friend's rack was affected by a (seven…

Were you writing to the MBR somehow when it died? If not, that looks like really badly designed hardware to me.

I've had abrupt shutdowns happen on laptops (one had a particularly loose battery...), desktops, servers and although have encountered corrupt files and filesystems, never had any of them corrupt the MBR.

Re: When should I not kill -9 a process?

#33
post #31
post #10

Earlier quoted context omitted.

You can't really corrupt a database that easily, can you? That's half the point of using a database, so you have transactions, etc.

I've never corrupted the DB, but I have ended up with inconsistencies because I needed to do several related queries in a row. Because I didn't send them as a single transaction from my application some of them were run, some not. Edit: Especially when you have multiple systems that should be consistent. Deleting a user from the DB, but being killed before it can be removed from our auth system, for instance.

The typical way of ensuring consistency in cases where the systems are unable to participate in a transaction would be to use a durable queue between the systems that can participate. So, in your Edit example, you would delete the user from the DB and add it the user to a queue in the same database to be deleted. Another process would then pull from that queue and attempt to delete from the auth system and only acknowledges the message once it is successful. You will still temporarily be in an inconsistent state but ultimately the operation will complete. This design only works for idempotent work.

Re: When should I not kill -9 a process?

#35
I think caution is still key, if you don't know what's going on Slow is Fast here. Several years ago while I was on an airplane flying to spend a nice vacation break with my family my admin partner tried to shutdown a MySQL db the "right way". He logged in and ran a mysqladmin shutdown and waited for a while. Not sure how long he waited but he claimed it was a "long time". Since it felt like there was no response to the command he assumed the database was hung and issued a kill -9 on all the mysql processes.

Sadly, what he failed to check was disk IO stats, this MySQL setup had heavy innodb table usage and settings that where deliberately set for more performance then reliability (large buffers, delayed commits etc.). What was going on was normal, MySQL was flushing everything to disk and to logs and was most likely going to stop without a problem.

He didn't look at the facts at hand, the disk IO was still going, MySQL was mostly writing to the log files, users where not being let in so the db was doing an orderly shutdown. Instead with the adrenalin pumping he felt he had “waited a long time” and issued the kill -9 and corrupted the InnoDB logs and tables beyond all recognition.

I landed at the airport to five frantic voicemails because this db was the core of a bunch of high profile sites and he was up to his ears in phone calls from the client. I had to spend the first 9 hours of my vacation with my kids playing in the background while I sweated it out on a laptop over a crappy connection that kept dropping me.

Yes I know, MySQL should have been able to handle the "power out" but this event was made worse because he started the shutdown, we had a deliberately fragile implementation, he didn't check the slaves so we didn't have a clean fall back and meanwhile he "waited a long time" but never checked the process to see what it was doing.

I use kill -9 (-KILL) all the time, but I do it where I know it's needed. Most of the time kill just works and if it doesn't that should give you pause to think carefully about what you'll do next. Slow is Fast and Fast is Slow, if you quickly do something radical like kill -9 or init 6 or 10 second power button crash then you may be spending the rest of your day cleaning up. Slow down a bit, look, listen and gather facts about the situation then make an informed decision. At least if you do all that and the rest of your day is still ruined you won't have that nagging feeling you shot yourself in the foot and you can talk intelligently to your client or boss about steps you took to avoid the situation.

My failure I suppose was that I hadn't explained to him that it was typical for the db to take upwards of 5 to 8 minutes to cleanly shutdown. Which gets to a second topic, documentation for production systems is essential, when the fire is on too many mistakes can be made because of "knowledge gaps" between team members. Needless to say after this incident I wrote extensive documentation for the team so the next time I was "on vacation" I could actually be "on vacation". :-)

Re: When should I not kill -9 a process?

#36
post #28
post #8

Earlier quoted context omitted.

This is a bit like saying that since your car can do emergency braking, you should always do emergency braking. Some processes clean up after themselves more neatly, or finish the current run of what they're doing first. It's dependent on what it is that you're stopping.

How is it anything like that?

"Use the harshest survivable option available, regardless of whether it's the best for the situation"

Re: When should I not kill -9 a process?

#37
post #8
post #2

Fiber optic cables are dug up by backhoes. Hard drives randomly fail. RAM is corrupted by cosmic rays. Racks lose power. CPU fans stop spinning. If your process relies on not being kill -9'd, then you might as well quit programming and go buy a lottery ticket.

This is a bit like saying that since your car can do emergency braking, you should always do emergency braking. Some processes clean up after themselves more neatly, or finish the current run of what they're doing first. It's dependent on what it is that you're stopping.

It seems more like saying "children, cyclists, animals, other drivers. If your car can't do emergency braking you might as well stop driving and buy a lottery ticket" to me. Which IMO is pretty reasonable. They don't say anything about when to use kill -9 (all examples are outside user control) just that it should be survivable.

Re: When should I not kill -9 a process?

#39
post #30

We write all of our server code with kill -9 in mind. Basically, eveything we have can be killed with -9 without any problems. It needs some cleanup code for lefover files or things like that. And use of atomic operations here and there.. But then you are ready for all kinds of hardware issues.

Are there resources on how to deal with this that you'd recommend? Is it even an issue in a higher level programming language or will the issue be abstracted away by a language above say, C?

Re: When should I not kill -9 a process?

#40
post #15

Earlier quoted context omitted.

kill -9 should never be worse than pulling the power plug No, it shouldn't, but just pulling the power plug isn't exactly recommended behaviour, either. There aren't many admins out there who will happily yank the power cord out of their desktops when they want to power it down. Last night I spent several hours getting a server back into gear after a 'pulled power plug' event. A friend's rack was affected by a (seven…

Were you writing to the MBR somehow when it died? If not, that looks like really badly designed hardware to me. I've had abrupt shutdowns happen on laptops (one had a particularly loose battery...), desktops, servers and although have encountered corrupt files and filesystems, never had any of them corrupt the MBR.

It's unclear what caused the MBR problems - the power outage was a couple of days before, and the system seemed to come back okay. My friend was busy with work and a cursory check had it clear, but yesterday things started acting funny, he logged in and the load was 40 and rising before it became unresponsive to his diagnostics. This machine had been running happily for quite some time before the powerout event (it's basically just a kvm host, the fun stuff is on the guests), so it's particularly puzzling. Somehow the MBR was overwritten with a syslinux one, and he says that box had never had syslinux used on it (extlinux, yes). The root cause will become evident at some point, it just needs some head-scratching time.
Post reply on HN