Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#11

Post by c00p3r »

The obvious idea (for admins) is to convert the setup to nginx + php 5.3.3 in fpm-mode + eaccelerator. (fpm patches are already part of the php distribution as from 5.3.2)

There are millions of step-by-step tutorials for these keywords.

If you have a test installation you can do it yourself and then change settings on your production server. If not you should setup an another copy on the same server with a different port nginx should listen to and empty drupal instance, test it. Then you can stop apache and start nginx on a correct port with proper drupal instance.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#12

Post by nl »

brandnewlow wrote: post #3
spooneybarger wrote: post #2

I'm not a drupal person. Have very little experience with it, but I take a perverse joy in tackling problems like this. Go figure. I assume from the description that apache is running mod_php and that is how drupal comes into the apache crashing part? how exactly is it crashing? can you give a few details about any errors in the logs, a more detailed description of what a 'crash' is etc?

The last time I looked under the hood, I observed this behavior. 1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way. 2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads. 3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been chec…

2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads.

To me this indicates some pages loads are blocking. I've seen this before where page loads triggered blocking network calls (web services) and the web service client code wasn't configured to use timeouts.

I'd take a look for that.

Another thing you can do is to treat the symptoms. Setup a cron job that runs every X minutes and kills the Y oldest threads.

From memory there is a config setting in Apache that specifies the number of times a thread should be reused, so try tweaking that, too.

(Edit: if this is the case, then switching to Ngnix won't solve it. You really need to understand why that threadcount is rising)

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#13

Post by brandnewlow »

c00p3r wrote: post #11

The obvious idea (for admins) is to convert the setup to nginx + php 5.3.3 in fpm-mode + eaccelerator. (fpm patches are already part of the php distribution as from 5.3.2) There are millions of step-by-step tutorials for these keywords. If you have a test installation you can do it yourself and then change settings on your production server. If not you should setup an another copy on the same server with a different…

We are running Nginx to serve all our anonymous traffic.

It works great. We've been front-paged on Digg and REddit in the last few months and the site didn't even flinch. You could drop a bomb on us and Nginx would behave like a champ.

But then we still crash every 80 minutes or so no matter how many or how few visitors we have to the site.

So nginx is awesome and we're using it bigtime and with great results. Not solving this one though I don't think.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#15

Post by mkramlich »

I wish you the best of luck in solving this problem but I also want to say I really don't want this kind of post to become a trend on HN. Really bad venue for it, so many better places. We don't want this to turn into the software developer's equivalent of, "Could you guys do my homework for me?" like so many other sites and forums have become.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#16

Post by c00p3r »

brandnewlow wrote: post #13
c00p3r wrote: post #11

The obvious idea (for admins) is to convert the setup to nginx + php 5.3.3 in fpm-mode + eaccelerator. (fpm patches are already part of the php distribution as from 5.3.2) There are millions of step-by-step tutorials for these keywords. If you have a test installation you can do it yourself and then change settings on your production server. If not you should setup an another copy on the same server with a different…

We are running Nginx to serve all our anonymous traffic. It works great. We've been front-paged on Digg and REddit in the last few months and the site didn't even flinch. You could drop a bomb on us and Nginx would behave like a champ. But then we still crash every 80 minutes or so no matter how many or how few visitors we have to the site. So nginx is awesome and we're using it bigtime and with great results. Not so…

The idea isn't about nginx itself, but of using php in an fpm mode along with eaccelerator.

In an fpm more php runs as a standalone FastCGI-server, outside of apache or other web server. It could run on the same machine with a web server (in that case you should configure them to use unix-sockets, which is even better) or on a separate one.

You will save a lot of memory, and will gain even more speed due to php script's caching.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#17

Post by oomkiller »

You need to trash Apache/mod_php. The little PHP I have done has been with Nginx (which it seems you are already using), and FastCGI. It may not be as automagic as mod_php, but since I'm a Rails developer/deployer, I'm used to it. Get a couple of FCGI processes running, set up and nginx upstream to talk to them, and set nginx to direct the traffic to them. Also, you can use something like God or Bluepill (I use the latter) to monitor these processes and reboot them if they lock up. It seems monit would also work well for you. The key thing here is the monitoring, as you can restart processes if they lock, and nginx will just skip over that port and use another process that is not locked/restarting. Finally, if you don't understand any of this, my info is on my profile, and we might be able to help you with Nginx or migrate your site to Rails ;)
Post reply on HN