Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin • Hacker News Board

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#21

Post by chrisbolt » Wed, Sep 01, 2010, 6:36 AM UTC

brandnewlow wrote: post #3

spooneybarger wrote: post #2

I'm not a drupal person. Have very little experience with it, but I take a perverse joy in tackling problems like this. Go figure. I assume from the description that apache is running mod_php and that is how drupal comes into the apache crashing part? how exactly is it crashing? can you give a few details about any errors in the logs, a more detailed description of what a 'crash' is etc?

The last time I looked under the hood, I observed this behavior. 1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way. 2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. 3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been chec…

2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads.

Threads or processes? If using the worker mpm, switch to prefork.

If problems persist, strace a frozen process to see what it's doing. strace is an incredibly useful diagnostic tool.

You could even set up a script to parse the server-status page, find out which processes are hung based on the SS column (seconds since beginning of most recent request) and kill that PID... but that's not pretty.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#22

Post by Jlambert » Wed, Sep 01, 2010, 6:36 AM UTC

Um... I own one of the drupal shops here in the us. Drop me an email: j@workhabit.com... I'm sure I can help you figure it out. Sounds like a config problem.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#23

Post by Jlambert » Wed, Sep 01, 2010, 6:37 AM UTC

Um... I own one of the drupal shops here in the us. Drop me an email: j@workhabit.com... I'm sure I can help you figure it out. Sounds like a config problem.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#24

Post by brandnewlow » Wed, Sep 01, 2010, 6:40 AM UTC

dpcan wrote: post #20

gaius wrote: post #18

By "clockwork" you mean, every hour, on the hour? That points to something running out of cron...

Like log file rotations of some kind.

Had 16 Apache freezes today, each about 80-90 minutes apart.

They happen late at night when we don't get hardly any visitors and during the day when we get lots. Due to nginx though, I don't think most of our visitors are being passed through into apache.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#25

Post by paulbaumgart » Wed, Sep 01, 2010, 6:53 AM UTC

mkramlich wrote: post #15

I wish you the best of luck in solving this problem but I also want to say I really don't want this kind of post to become a trend on HN. Really bad venue for it, so many better places. We don't want this to turn into the software developer's equivalent of, "Could you guys do my homework for me?" like so many other sites and forums have become.

He's a contributing member of the community and clearly has put a lot of effort into solving the problem himself. I think as long as you've reached some adequately high threshold of desperation, a post like this is perfectly fine.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#26

Post by valentin » Wed, Sep 01, 2010, 6:55 AM UTC

brandnewlow wrote: post #24

dpcan wrote: post #20

Earlier quoted context omitted.

Like log file rotations of some kind.

Had 16 Apache freezes today, each about 80-90 minutes apart. They happen late at night when we don't get hardly any visitors and during the day when we get lots. Due to nginx though, I don't think most of our visitors are being passed through into apache.

It might be a memory leak. You can lower the maxrequestsperchild in order to respawn the childrens.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#27

Post by gaius » Wed, Sep 01, 2010, 7:00 AM UTC

brandnewlow wrote: post #24

dpcan wrote: post #20

Earlier quoted context omitted.

Like log file rotations of some kind.

Had 16 Apache freezes today, each about 80-90 minutes apart. They happen late at night when we don't get hardly any visitors and during the day when we get lots. Due to nginx though, I don't think most of our visitors are being passed through into apache.

That's not what you said in your problem specification!

The key to solving any issue is to precisely and unambiguously define exactly what is (or is not) happening and how this deviates from the norm.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#28

Post by brandnewlow » Wed, Sep 01, 2010, 7:01 AM UTC

gaius wrote: post #27

brandnewlow wrote: post #24

Earlier quoted context omitted.

Had 16 Apache freezes today, each about 80-90 minutes apart. They happen late at night when we don't get hardly any visitors and during the day when we get lots. Due to nginx though, I don't think most of our visitors are being passed through into apache.

That's not what you said in your problem specification! The key to solving any issue is to precisely and unambiguously define exactly what is (or is not) happening and how this deviates from the norm.

Very fair. Duly noted.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#29

Post by po » Wed, Sep 01, 2010, 7:14 AM UTC

Here's what your problem is:

One day, your software had a resource leak and was eating up too much of memory/file handles/etc... One of your developers tried restarting the system and like magic, it fixed it. After a few days of having to do this, he got tired and simply put the restart command into an hourly cron job or a script somewhere on the system.

I've seen it a thousand times. My consulting rate is $200 an hour. If this turns out to be the problem, I'll expect $6.67 in the mail from you. :-)

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#30

Post by SkyMarshal » Wed, Sep 01, 2010, 7:27 AM UTC

paulbaumgart wrote: post #25

mkramlich wrote: post #15

I wish you the best of luck in solving this problem but I also want to say I really don't want this kind of post to become a trend on HN. Really bad venue for it, so many better places. We don't want this to turn into the software developer's equivalent of, "Could you guys do my homework for me?" like so many other sites and forums have become.

He's a contributing member of the community and clearly has put a lot of effort into solving the problem himself. I think as long as you've reached some adequately high threshold of desperation, a post like this is perfectly fine.

Agreed. He's also clearly not asking for a free solution to his homework, but to hire someone to fix the problem as freelance contract, via his last paragraph.