Hire HN: Apache is crashing every hour. Seeking Drupal/Nginx-friendly sysadmin

Very close. From what he later posts about supervisord monitoring - he is basically hitting a server bottleneck of number of threads, which causes him to run out of memory and get into swap space. Then everything goes to hell and he (supervisord) has to restart.

Proof: You will have 0 memory free and start using swap when the restart happens - type 'top' and watch this go down.

Fix: Use less threads. The number of concurrent users you can support is the # of pages you can serve up per second in each thread times the number of threads your server can handle. So a huge improvement can be caching. Cache pages, blocks, hell maybe your whole site. The cache hits will serve 10-100 times more responses per second than generating your entire page.

I consulted on a project that was having horrible server performance on a huge server with relatively static data. The way they produced pages was really poor and took 3 or 4 seconds. There is no way you can serve up a large number of concurrent requests with the turnaround time over a second. I added caching as a quick fix and they ended up just being happy that 99.9% of their pages served up in 1/100th of a second.

I second this. I had a similar problem before, and yes I mis-configured my apache. In my case it was random crashes instead of hourly.

I would suggest you to post your problem to your hosting community forum with related info (logs, conf files, server setup info, etc). In my case, it's the Linode forum. I'm sure the community is more than happy to assist you.

Good luck!

Instead of watching it in 'top', I recommend installing munin: http://munin-monitoring.org/

The blurb from that site:

Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.

If there is anything viaweb has taught me, nothing beats performance like static html pages.

Can you publish static html files and simply have an ajax component for the commenting and any other dynamic stuff handled by node.js?

Just throwing that idea out there. Criticisms?

Yeah, I know he is just looking to get his existing setup stable, and he shouldn't change it.

(raises hand) my turn! (I fix these problems and other stuff for a living).

There are short answers and long answers to this problem.

The short answers try to fix quickly what the most likely cause of the problem is. If a server is being unresponsive this is most likely because the server runs out of memory or incurs in heavy disk I/O. For the memory problem find what are the processes eating up the memory. For apache lowering the MaxClients/MaxServers/KeepAlive helps. For I/O you'll have to look at the database (see error log, log long queries etc).

But that's speculation.

The proper way to fix the issue is to understand what's really causing it. A way to start is getting meaningful server utilization data and reviewing logs.

Spend a few minutes installing a graphical monitoring tool like munin or google's quicklook (I like these because they are lightweight, there's a ton of them like cacti etc). In a couple of hours you'll get a better picture of what's going on.

Review the logs, specially error logs.

I use a little script that runs every 15 mins and logs the most important stuff that is going on in the server, you can run something similar every 5 mins or whatever to find what's really going on: http://pastebin.com/Qv0J3WHY

Another tip is to set up a mirror test/stage server and benchmark the heck out of it; you don't want to make a lot of changes live in a production server.

I guess you can also ask the support people at Slicehost.

I'll gladly take a look at your server for up to 30 mins at no cost.