Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#31

Post by samuel »

c00p3r wrote: post #11

The obvious idea (for admins) is to convert the setup to nginx + php 5.3.3 in fpm-mode + eaccelerator. (fpm patches are already part of the php distribution as from 5.3.2) There are millions of step-by-step tutorials for these keywords. If you have a test installation you can do it yourself and then change settings on your production server. If not you should setup an another copy on the same server with a different…

I wouldn't recommend to blindly change his setup. This is the sysadmin equivalent of changing lines of code almost randomly to fix a bug.

When there is a problem the very first step is to find out what's happening. Apache is rock solid software so there must be something in his setup or software that isn't right.

My line of action would be:

  - check the logs(apache and system ones), looking for errors or limits reached.
  - if no clue, try to reproduce it, same os, software versions. Make scripts to simulate the load.
  - strace the processes when they lockup (dtrace 'em in Solaris).
  - get a coredump(kill -SIGABRT), compile apache and their module with symbols (-g CFLAG), an gdb it.
I'm not a sysadmin anymore, but that's what I would have done when I was one.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#32

Post by gojomo »

Good news! Crashing hourly is better than a more erratic schedule; it means you can observe the crash nearly on demand.

I would disable the auto-restart for a couple cycles to get a closer look at the unresponsive condition: total threads, any swapping, any busy threads. It means a few more minutes downtime but more information for you.

It sounds like some connections are hanging indefinitely, eventually using up 'all' of some capped resource (threads, RAM, etc.).

Make sure you have a 'Timeout' setting in the applicable apache conf, and make it really small (10-20) to see if that helps. (You may get complaints that other long-running-requests that used to finish now fail -- but at least service will remain available for usual requests, and you can adjust the value back up later.)

Check the end of the error_log just after a crash/freeze for hints. Consider adjusting the conf value MaxClients up, as long as there was no swapping evident. Consider adjusting the conf value MaxRequestsPerChild down, but nonzero, so that children are recycled sooner before they grow problematic from memory leaks.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#33

Post by buro9 »

This sounds like a simple mis-configuration of Apache and the maxrequestsperchild.

If Apache is spawning 200 processes, each of which have PHP within that, then you can easily run out of memory and then kill apache and the system.

It looks like, from your hardware specs, that you've had this problem for a while and chose to solve it by throwing hardware at the problem... gradually increasing the RAM of the VM until you have a 4GB web server.

My bet is that this 4GB isn't needed, and that if you had got someone with some understanding of apache to look at it sooner that they could've configured apache to have fewer child processes and this would avoid the heavy memory requirements.

Go check serverfault.com, possibly hire someone to look at the problem for you if you don't understand what you're doing here. You will need to provide real information such as your apache2.conf and php.ini files, as well as info on whether you're using things like XCache, Memcache (if so, where is this installed?), Varnish, etc.

For your traffic and hardware, the numbers look low. You're most likely just running a badly configured apache.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#34

Post by 16s »

po wrote: post #29

Here's what your problem is: One day, your software had a resource leak and was eating up too much of memory/file handles/etc... One of your developers tried restarting the system and like magic, it fixed it. After a few days of having to do this, he got tired and simply put the restart command into an hourly cron job or a script somewhere on the system. I've seen it a thousand times. My consulting rate is $200 an ho…

I have actually done that in a test environment (never production). The only difference was my cron job was every 15 minutes. Doing that fixed an authentication issue with Apache and Microsoft Active Directory well enough for Devs to authenticate without getting errors.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#35

Post by c00p3r »

samuel wrote: post #31
c00p3r wrote: post #11

The obvious idea (for admins) is to convert the setup to nginx + php 5.3.3 in fpm-mode + eaccelerator. (fpm patches are already part of the php distribution as from 5.3.2) There are millions of step-by-step tutorials for these keywords. If you have a test installation you can do it yourself and then change settings on your production server. If not you should setup an another copy on the same server with a different…

I wouldn't recommend to blindly change his setup. This is the sysadmin equivalent of changing lines of code almost randomly to fix a bug. When there is a problem the very first step is to find out what's happening. Apache is rock solid software so there must be something in his setup or software that isn't right. My line of action would be: - check the logs(apache and system ones), looking for errors or limits reache…

good luck in debugging mpm_worker with all loaded modules and their dependencies. btw, who will pay for hours and hours of such 'practicing'?

Is that never crossed your mind that the solution I recommended were battle-tested one?

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#36

Post by ritonlajoie »

I'm a little embarrassed to see so many guys telling brandnewlow to change his configuration in such a situation. If my understanding is well, then he is on an urgent situation and you are basically telling him to take risks in having a completely new setup ? (removing apache/mod_php and puttin nginx and stuff).

Seriously guys... brandnewlow, please don't read these advices. Maybe on the short term you can think about changing your setup. But right now, listen to those who try to fix your _current_ setup !

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#37

Post by samuel »

c00p3r wrote: post #35
samuel wrote: post #31

Earlier quoted context omitted.

I wouldn't recommend to blindly change his setup. This is the sysadmin equivalent of changing lines of code almost randomly to fix a bug. When there is a problem the very first step is to find out what's happening. Apache is rock solid software so there must be something in his setup or software that isn't right. My line of action would be: - check the logs(apache and system ones), looking for errors or limits reache…

good luck in debugging mpm_worker with all loaded modules and their dependencies. btw, who will pay for hours and hours of such 'practicing'? Is that never crossed your mind that the solution I recommended were battle-tested one?

That's the last resort: almost never you need to get that far(stracing/trussing usually do the trick), and it doesn't have to take "hours and hours" if you have done it before, specially if it's a locking issue. I don't remember if I have had to gdb apache(I think I have), but sendmail + milters for sure.

I don't doubt that your solution has been "battle-tested", but I assure you that apache + mod_php also has. Now if he has a different problem with that new setup what you will recommend then? Change again to cherokee or lighttpd? Or do you really think that nginx + fpm is a silver bullet?

I accept that spending some time cheap shooting randomly(raising os limits, changing threading model, upgrading to new version...) may be worth the time, though. But changing completely your middleware isn't a cheap shot in my book.

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#38

Post by roel_v »

c00p3r wrote: post #11

The obvious idea (for admins) is to convert the setup to nginx + php 5.3.3 in fpm-mode + eaccelerator. (fpm patches are already part of the php distribution as from 5.3.2) There are millions of step-by-step tutorials for these keywords. If you have a test installation you can do it yourself and then change settings on your production server. If not you should setup an another copy on the same server with a different…

We used to run a site with an order of magnitude more visitors on a couple of Pentium III's, over 10 years ago, on Apache and 2.0 something Linux kernels. PHP 3, too. Now I guess that the memory requirements have gone up in those years, but still I find it hard to believe that the current setup can't be made to work. Switching to prefork seems the first obvious step. I haven't done real sysadmin work since then though so what do I know ;)

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#39

Post by whimsy »

mkramlich wrote: post #15

I wish you the best of luck in solving this problem but I also want to say I really don't want this kind of post to become a trend on HN. Really bad venue for it, so many better places. We don't want this to turn into the software developer's equivalent of, "Could you guys do my homework for me?" like so many other sites and forums have become.

Were the first two words in the title not "Hire HN" when you posted this comment?

Re: Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

#40

Post by c00p3r »

samuel wrote: post #37
c00p3r wrote: post #35

Earlier quoted context omitted.

good luck in debugging mpm_worker with all loaded modules and their dependencies. btw, who will pay for hours and hours of such 'practicing'? Is that never crossed your mind that the solution I recommended were battle-tested one?

That's the last resort: almost never you need to get that far(stracing/trussing usually do the trick), and it doesn't have to take "hours and hours" if you have done it before, specially if it's a locking issue. I don't remember if I have had to gdb apache(I think I have), but sendmail + milters for sure. I don't doubt that your solution has been "battle-tested", but I assure you that apache + mod_php also has. Now i…

Re-read the OP - the problem is with threads. Switch apache to a prefork mpm - is a good solution, but replace apache with simple and more efficient setup is a much better solution.

nginx + fpm isn't a silver bullet, but it is much more efficient an flexible solution. And avoiding using threads with such a terrible mess as PHP is also good solution, which will save you a lot of time and effort.

Just think about how many things will be dynamically loaded, and how much of this code is really thread-safe and properly tested.

btw, I bet there is a mere sigsegv.

Post reply on HN