Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When should I not kill -9 a process?

unix.stackexchange.com

Post reply on HN
Print view View original

41–50 of 85 posts

Re: When should I not kill -9 a process?

#41

Post by jeffdavis »

kator wrote: post #35

I think caution is still key, if you don't know what's going on Slow is Fast here. Several years ago while I was on an airplane flying to spend a nice vacation break with my family my admin partner tried to shutdown a MySQL db the "right way". He logged in and ran a mysqladmin shutdown and waited for a while. Not sure how long he waited but he claimed it was a "long time". Since it felt like there was no response to…

+1.

Postgres is designed to be resilient to "kill -9" as well as hard power offs. Even if you use durability-sacrificing features like asynchronous commit[1] or unlogged tables[2], the risks are very well-defined and contained to recent transactions and data in that unlogged table, respectively.

But even for postgres, you have to be a bit careful. For instance, many disk drives lie about completing the writes and really have them in a volatile cache, so a hard power off can still cause corruption. You need to disable the write cache on the disks using hdparm (or similar) to be safe. And "kill -9" is quite annoying, because the child processes don't have a good way to know that the parent is exiting, so then you will be unable to start the new parent process until the children have all exited as well.

EDIT: There's still no excuse for MySQL completely corrupting the system on a kill -9. That's just a misdesign -- consider that the out-of-memory (OOM) killer on linux sends a -9.

[1] http://www.postgresql.org/docs/9.4/static/runtime-config-wal...

[2] http://www.postgresql.org/docs/9.4/static/sql-createtable.ht...

Re: When should I not kill -9 a process?

#42

Post by userbinator »

vacri wrote: post #40
userbinator wrote: post #32

Earlier quoted context omitted.

Were you writing to the MBR somehow when it died? If not, that looks like really badly designed hardware to me. I've had abrupt shutdowns happen on laptops (one had a particularly loose battery...), desktops, servers and although have encountered corrupt files and filesystems, never had any of them corrupt the MBR.

It's unclear what caused the MBR problems - the power outage was a couple of days before, and the system seemed to come back okay. My friend was busy with work and a cursory check had it clear, but yesterday things started acting funny, he logged in and the load was 40 and rising before it became unresponsive to his diagnostics. This machine had been running happily for quite some time before the powerout event (it's…

I almost read your last sentence as "The root kit will become evident at some point", because that's what came to mind with those symptoms. I'd check for an infection.

Re: When should I not kill -9 a process?

#43

Post by jeffdavis »

To anyone saying that you shouldn't "kill -9" a process, or that you should do some song-and-dance first: kill -9 is exactly what the OOM (out-of-memory) killer on linux does when memory is short. Typically, the application has no good way to even know that memory is short, because linux radically overcommits memory and still won't return a NULL from a malloc().

So, software should be written to assume it might be killed if you want to have a robust system.

By the way, using a small fixed amount of memory is no defense, or at least not in all kernel versions. The "badness" heuristic function used to find the victim could end up counting the same byte of memory many times:

http://thoughts.davisjeff.com/2009/11/29/linux-oom-killer/

Re: When should I not kill -9 a process?

#45

Post by jeffdavis »

IsTom wrote: post #17
btmorex wrote: post #4

As if often the case with stackoverflow answers, all of them are wrong in different ways. You should only kill -9 when every other signal the program is likely to respond to has not worked. kill -9 is likely to leave program in a state that requires manual intervention, especially if that program is a database. If you're a developer, before you kill -9 a program send SIGTERM (ie kill without args or kill -15). If the…

If a database is not kill -9 resistant then how can it be power-off resistant?

There are ways to misdesign a system such that it is power-off robust but not kill -9 robust. The reason is that power-off means everything dies.

Postgres has special code to account for hard kills: it has a SYSV shared memory segment, to which every child attaches. If the parent dies, the children don't have a good way to know, so they might keep running. If you try to start a new postmaster and it sees that there are still processes attached to the shared memory segment (shm_nattch), it will fail to start.

Were it not for that code, it would allow you to start a new parent process, leading to chaos as two sets of backends were accessing the same files and shared memory without knowing about eachother.

Re: When should I not kill -9 a process?

#46

Post by jeffdavis »

justizin wrote: post #7

Over 15 years ago, as a teenager, I taught Linux / UNIX Admin courses, and worked as a consultant advising folks, and in the late 90s I was very adamant that you should never -9 anything unless you know exactly what you are doing. As infrastructures have grown, and I have managed large applications involving tens to hundreds, often over a thousand servers, and I have grown to accept that a power supply can fail and a…

"kill -9 should never be worse than pulling the power plug"

It's actually easy to misdesign a system such that it is safe against power failures but not kill -9. See my other comment:

https://news.ycombinator.com/item?id=7793301

Re: When should I not kill -9 a process?

#47

Post by al_gore »

mcintyre1994 wrote: post #37
vacri wrote: post #8

Earlier quoted context omitted.

This is a bit like saying that since your car can do emergency braking, you should always do emergency braking. Some processes clean up after themselves more neatly, or finish the current run of what they're doing first. It's dependent on what it is that you're stopping.

It seems more like saying "children, cyclists, animals, other drivers. If your car can't do emergency braking you might as well stop driving and buy a lottery ticket" to me. Which IMO is pretty reasonable. They don't say anything about when to use kill -9 (all examples are outside user control) just that it should be survivable.

I get what you're going for, but it's pretty much openly legal to kill pedestrians with cars (in the US, at least). "No criminality suspected"!

Re: When should I not kill -9 a process?

#48

Post by falcolas »

jeffdavis wrote: post #41
kator wrote: post #35

I think caution is still key, if you don't know what's going on Slow is Fast here. Several years ago while I was on an airplane flying to spend a nice vacation break with my family my admin partner tried to shutdown a MySQL db the "right way". He logged in and ran a mysqladmin shutdown and waited for a while. Not sure how long he waited but he claimed it was a "long time". Since it felt like there was no response to…

+1. Postgres is designed to be resilient to "kill -9" as well as hard power offs. Even if you use durability-sacrificing features like asynchronous commit[1] or unlogged tables[2], the risks are very well-defined and contained to recent transactions and data in that unlogged table, respectively. But even for postgres, you have to be a bit careful. For instance, many disk drives lie about completing the writes and rea…

You, and the OP, are likely mistaken about the cause of the corruption. MySQL, running InnoDB, is ACID compliant. InnoDB takes that durability seriously, and even by hand-tuning the performance factors, it's very hard to put InnoDB in a state where a simple process death, even during shutdown, will corrupt the files on disk.

If the database truly corrupted only due to improper shutdown, it was because the double write buffers were disabled on a non-atomic FS in which case you're intentionally risking DB corruption. More likely, there was bit-rot which was undetected during the normal running state of MySQL, and recovery couldn't get around it.

On the anecdote side, I've yet to have MySQL corrupt a database from a kill -9, and as I do a lot of failover development and testing, I'm issuing kill -9 to running databases frequently.

Re: When should I not kill -9 a process?

#49

Post by falcolas »

jeffdavis wrote: post #43

To anyone saying that you shouldn't "kill -9" a process, or that you should do some song-and-dance first: kill -9 is exactly what the OOM (out-of-memory) killer on linux does when memory is short. Typically, the application has no good way to even know that memory is short, because linux radically overcommits memory and still won't return a NULL from a malloc(). So, software should be written to assume it might be ki…

Just because a program which is designed to prevent kernel panics due to OOM kill -9s a process, doesn't mean that you as a sysadmin should.

kill -15 typically leaves processes in a properly shut down state, which in terms of databases alone means that they will start up without a recovery process (which can be a 20-30 minute operation sometimes). That alone makes waiting a few minutes for a running process to respond to a kill -15 worthwhile.

Re: When should I not kill -9 a process?

#50

Post by falcolas »

oleganza wrote: post #14

If the process is not designed to survive the crash than it's a more like a bug. I'd rather encourage everyone to design programs in a robust way: when they can clean up after themselves upon relaunch.

And what about when that cleanup takes longer than a proper shutdown?
Post reply on HN