Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 54 posts

Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#1

Post by brandnewlow »

I run a social news site called Windy Citizen in Chicago. Right now and for some time, it has been majorly, and embarrassingly screwed up. It crashes every hour, like clockwork.

I first reached out to my network of developers I've hired in the past for contract work to fix the problem. Three of them either a) couldn't figure it out or b) got bored working on it and said they didn't want to spend any more time on it.

I then asked a few gracious dev friends for advice. This got me some theories and ideas that I've spent a great deal of time trying to prove out. I have not been successful though. We keep crashing hourly.

So I turn to you, HN to see if there's anyone on here who'd be interested in fixing this problem for our company. We're a two-man business, ad-supported and ramen-profitable, that gives people in Chicago a central place to share and talk about their favorite local stories.

This has been going on for close to two months though, and our users are getting sick of it. I'm doing a bad job taking care of them by not getting this fixed but am running out of options and people to ask about working on this.

I realize debugging the hosting set up on a complex Drupal site isn't the sexiest type of freelance gig to take on, but if there's interest in this, I will provide more detail in the comments and also explain why this is in fact a sexy job to take on.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#2

Post by spooneybarger »

I'm not a drupal person. Have very little experience with it, but I take a perverse joy in tackling problems like this. Go figure.

I assume from the description that apache is running mod_php and that is how drupal comes into the apache crashing part? how exactly is it crashing? can you give a few details about any errors in the logs, a more detailed description of what a 'crash' is etc?

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#3

Post by brandnewlow »

spooneybarger wrote: post #2

I'm not a drupal person. Have very little experience with it, but I take a perverse joy in tackling problems like this. Go figure. I assume from the description that apache is running mod_php and that is how drupal comes into the apache crashing part? how exactly is it crashing? can you give a few details about any errors in the logs, a more detailed description of what a 'crash' is etc?

The last time I looked under the hood, I observed this behavior.

1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way.

2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads.

3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been checking every 5 seconds to make sure apache is ok, spots this, and restarts apache. The site shows a fail-whale image for 5-10 seconds as SupervisorD does its thing. Then apache comes back online and we repeat the cycle.

4. About every 3 days, the semaphores pile up, and SupervisorD is unable to restart apache. Apache goes into FATAL mode and I have to run a funky command line command a friend showed me to clear out the semaphores so Apache can be restarted and taken out of its tailspin.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#4

Post by spooneybarger »

brandnewlow wrote: post #3
spooneybarger wrote: post #2

I'm not a drupal person. Have very little experience with it, but I take a perverse joy in tackling problems like this. Go figure. I assume from the description that apache is running mod_php and that is how drupal comes into the apache crashing part? how exactly is it crashing? can you give a few details about any errors in the logs, a more detailed description of what a 'crash' is etc?

The last time I looked under the hood, I observed this behavior. 1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way. 2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. 3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been chec…

Is there a reason you are using apache in a threaded mode rather than pre-fork? Basically... is threaded a requirement or a choice? If a choice, have you tried running in pre-fork and if yes, do you get the same issues?

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#5

Post by brandnewlow »

spooneybarger wrote: post #4
brandnewlow wrote: post #3

Earlier quoted context omitted.

The last time I looked under the hood, I observed this behavior. 1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way. 2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. 3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been chec…

Is there a reason you are using apache in a threaded mode rather than pre-fork? Basically... is threaded a requirement or a choice? If a choice, have you tried running in pre-fork and if yes, do you get the same issues?

I did not set up our hosting environment, a developer friend of mine did, so I can't answer that first question. I don't know if Drupal requires threaded or not or how to try running it in pre-fork mode.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#6

Post by spooneybarger »

brandnewlow wrote: post #5
spooneybarger wrote: post #4

Earlier quoted context omitted.

Is there a reason you are using apache in a threaded mode rather than pre-fork? Basically... is threaded a requirement or a choice? If a choice, have you tried running in pre-fork and if yes, do you get the same issues?

I did not set up our hosting environment, a developer friend of mine did, so I can't answer that first question. I don't know if Drupal requires threaded or not or how to try running it in pre-fork mode.

I would assume that it isnt a requirement but rather an attempt to allow apache to handle more w/ less memory usage. Apache is a memory pig especially when using mod_* modules and running with in pre-fork.

Can you describe as best as you know, your hosting environment?

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#7

Post by brandnewlow »

brandnewlow wrote: post #5
spooneybarger wrote: post #4

Earlier quoted context omitted.

Is there a reason you are using apache in a threaded mode rather than pre-fork? Basically... is threaded a requirement or a choice? If a choice, have you tried running in pre-fork and if yes, do you get the same issues?

I did not set up our hosting environment, a developer friend of mine did, so I can't answer that first question. I don't know if Drupal requires threaded or not or how to try running it in pre-fork mode.

My hosting environment:

The site is hosted at Slicehost, on two slices.

Webserver: 4GB slice. DB server: 2GB slice.

We do about 3-5k uniques/day.

I've tried scaling the DB server up to 4GB to see if that solved the problem, but we continued crashing every hour.

Part of the problem though with my own efforts is that I don't have much experience with sysadmin stuff to be able to say for sure that any of my own observations and attempts to solve were done properly. For instance, I scaled up the slice, but maybe I didn't restart Apache in the right way afterward to know if that made a difference or not.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#8

Post by spooneybarger »

brandnewlow wrote: post #7
brandnewlow wrote: post #5

Earlier quoted context omitted.

I did not set up our hosting environment, a developer friend of mine did, so I can't answer that first question. I don't know if Drupal requires threaded or not or how to try running it in pre-fork mode.

My hosting environment: The site is hosted at Slicehost, on two slices. Webserver: 4GB slice. DB server: 2GB slice. We do about 3-5k uniques/day. I've tried scaling the DB server up to 4GB to see if that solved the problem, but we continued crashing every hour. Part of the problem though with my own efforts is that I don't have much experience with sysadmin stuff to be able to say for sure that any of my own observat…

hn thread isnt the best way to try and ask all these questions. i sent you an email at the contact address listed in your profile.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#10

Post by whalesalad »

Get rid of mod_php, replace it with a nice build of php-fpm (included in 5.3 actually) tied to nginx via fastcgi. Bam. Nginx will handle all static content SUPER fast while keeping an incredibly small memory footprint, while passing dynamic requests to the PHP side of things via fastcgi.

Example of one of my nginx configuration files, for a Wordpress site: http://paste.pocoo.org/show/256982/

This will only do so much, but I suspect it will solve a great deal of your problems. The final bit of advice that I can give is to look into any caching modules for Drupal. Simple disk caching will do the trick, just something to prevent lots of DB reads. I'm unfamiliar with Drupal, but I'd imagine there are loads of modules out there that can assist with caching.

Post reply on HN