Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When should I not kill -9 a process?

unix.stackexchange.com

Post reply on HN
Print view View original

51–60 of 85 posts

Re: When should I not kill -9 a process?

#51

Post by seunosewa »

btmorex wrote: post #20
TillE wrote: post #10

Earlier quoted context omitted.

You can't really corrupt a database that easily, can you? That's half the point of using a database, so you have transactions, etc.

not corrupt != everything is peachy At the very least, you need to be prepared for a possibly very long replay of logs. Also, a huge number of people run databases in a configuration that doesn't make those guarantees. For example, many people will run mysql (esp. less performant slaves) with innodb_flush_log_at_trx_commit = 0 for performance with the understanding that a failure might require manual fixes.

A failure with that parameter will only cause transactions committed in the last second to be lost. The DB won't require manual fixes.

Re: When should I not kill -9 a process?

#52

Post by kator »

falcolas wrote: post #48
jeffdavis wrote: post #41

Earlier quoted context omitted.

+1. Postgres is designed to be resilient to "kill -9" as well as hard power offs. Even if you use durability-sacrificing features like asynchronous commit[1] or unlogged tables[2], the risks are very well-defined and contained to recent transactions and data in that unlogged table, respectively. But even for postgres, you have to be a bit careful. For instance, many disk drives lie about completing the writes and rea…

You, and the OP, are likely mistaken about the cause of the corruption. MySQL, running InnoDB, is ACID compliant. InnoDB takes that durability seriously, and even by hand-tuning the performance factors, it's very hard to put InnoDB in a state where a simple process death, even during shutdown, will corrupt the files on disk. If the database truly corrupted only due to improper shutdown, it was because the double writ…

This was years ago and trust me it was corrupt. I think there may have been some issues with InnoDB in those days that made it a bit fragile in that situation.

Re: When should I not kill -9 a process?

#53

Post by falcolas »

kator wrote: post #52
falcolas wrote: post #48

Earlier quoted context omitted.

You, and the OP, are likely mistaken about the cause of the corruption. MySQL, running InnoDB, is ACID compliant. InnoDB takes that durability seriously, and even by hand-tuning the performance factors, it's very hard to put InnoDB in a state where a simple process death, even during shutdown, will corrupt the files on disk. If the database truly corrupted only due to improper shutdown, it was because the double writ…

This was years ago and trust me it was corrupt. I think there may have been some issues with InnoDB in those days that made it a bit fragile in that situation.

My apologies if it sounded like I was doubting that there was corruption - I was not. I was doubting that merely sending kill -9 to the process was the sole cause of the corruption. More than likely it uncovered the corruption due to having to do recovery against corrupted data.

Re: When should I not kill -9 a process?

#54

Post by oleganza »

falcolas wrote: post #50
oleganza wrote: post #14

If the process is not designed to survive the crash than it's a more like a bug. I'd rather encourage everyone to design programs in a robust way: when they can clean up after themselves upon relaunch.

And what about when that cleanup takes longer than a proper shutdown?

Good point. Can you give me an example?

Re: When should I not kill -9 a process?

#55

Post by spudlyo »

seunosewa wrote: post #51
btmorex wrote: post #20

Earlier quoted context omitted.

not corrupt != everything is peachy At the very least, you need to be prepared for a possibly very long replay of logs. Also, a huge number of people run databases in a configuration that doesn't make those guarantees. For example, many people will run mysql (esp. less performant slaves) with innodb_flush_log_at_trx_commit = 0 for performance with the understanding that a failure might require manual fixes.

A failure with that parameter will only cause transactions committed in the last second to be lost. The DB won't require manual fixes.

The DB won't require manual fixes.

Losing a transaction that was committed upstream can lead to a bunch of manual fixes in replicated database clusters. What happens if rows inserted in that lost transaction are eventually updated? Replication breaks, you get paged, and now you're faced with either manually fixing the DB consistency error or by rebuilding the whole slave. Woe be unto you if this host is a replication hub with a bunch of slaves hanging off it.

Re: When should I not kill -9 a process?

#57

Post by twistedpair »

jrockway wrote: post #2

Fiber optic cables are dug up by backhoes. Hard drives randomly fail. RAM is corrupted by cosmic rays. Racks lose power. CPU fans stop spinning. If your process relies on not being kill -9'd, then you might as well quit programming and go buy a lottery ticket.

I know of some PHP installs that add a cron job that kills off hung PHP processes every hour. Ain't that uncommmon.

Re: When should I not kill -9 a process?

#58

Post by jeffdavis »

falcolas wrote: post #49
jeffdavis wrote: post #43

To anyone saying that you shouldn't "kill -9" a process, or that you should do some song-and-dance first: kill -9 is exactly what the OOM (out-of-memory) killer on linux does when memory is short. Typically, the application has no good way to even know that memory is short, because linux radically overcommits memory and still won't return a NULL from a malloc(). So, software should be written to assume it might be ki…

Just because a program which is designed to prevent kernel panics due to OOM kill -9s a process, doesn't mean that you as a sysadmin should. kill -15 typically leaves processes in a properly shut down state, which in terms of databases alone means that they will start up without a recovery process (which can be a 20-30 minute operation sometimes). That alone makes waiting a few minutes for a running process to respon…

I was commenting more about how software should be written than what an admin should do.

For admins, I wouldn't fault them much for kill -9, I would fault the software more if it lead to anything more than an inconvenience. But sure, it's wise to use -15 or whatever as long as it works.

Re: When should I not kill -9 a process?

#59

Post by aaronem »

al_gore wrote: post #47
mcintyre1994 wrote: post #37

Earlier quoted context omitted.

It seems more like saying "children, cyclists, animals, other drivers. If your car can't do emergency braking you might as well stop driving and buy a lottery ticket" to me. Which IMO is pretty reasonable. They don't say anything about when to use kill -9 (all examples are outside user control) just that it should be survivable.

I get what you're going for, but it's pretty much openly legal to kill pedestrians with cars (in the US, at least). "No criminality suspected"!

There's a little more to the United States than the city of New York. I grant it is an uncommonly large city, but it's not that large.

Re: When should I not kill -9 a process?

#60

Post by ZeroGravitas »

mcintyre1994 wrote: post #39
eknkc wrote: post #30

We write all of our server code with kill -9 in mind. Basically, eveything we have can be killed with -9 without any problems. It needs some cleanup code for lefover files or things like that. And use of atomic operations here and there.. But then you are ready for all kinds of hardware issues.

Are there resources on how to deal with this that you'd recommend? Is it even an issue in a higher level programming language or will the issue be abstracted away by a language above say, C?

I googled for an old LWN article on "crash-only" software and found thus request for similar resources:

http://stackoverflow.com/questions/2405172/resources-about-c...

Post reply on HN