Sysadmin mistakes start-ups make

#1

Post by polvi » Tue, Oct 20, 2009, 5:28 PM UTC

Sysadmin mistakes start-ups make
cloudkick.com

Re: Sysadmin mistakes start-ups make

#2

Post by DrJokepu » Tue, Oct 20, 2009, 5:47 PM UTC

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

Re: Sysadmin mistakes start-ups make

#3

Post by tptacek » Tue, Oct 20, 2009, 5:48 PM UTC

This was a great article, but I ended it wondering whether they either (a) knew what a system call was (until the end, I thought maybe they meant a system() shell-out) or (b) realize how many system calls a vanilla request/response cycle incurs.

Re: Sysadmin mistakes start-ups make

#4

Post by patio11 » Tue, Oct 20, 2009, 5:50 PM UTC

If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application.

I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning.

Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently making to system(big_command_to_invoke_imagemagick) ?

Re: Sysadmin mistakes start-ups make

#5

Post by pquerna » Tue, Oct 20, 2009, 5:50 PM UTC

DrJokepu wrote: post #2

Here's one we made recently: Purchasing an array of hard drives (for storage servers) and not making sure that not all of them are from the same batch. Since they were made in the same batch, they had the same defects and when they failed, they failed one after each other in a very short interval. Since all of them failed, RAID didn't help, we had to restore the day-old offline backup.

Good lesson that RAID != Backup!

RAID5 by chance?

Re: Sysadmin mistakes start-ups make

#6

Post by aristus » Tue, Oct 20, 2009, 5:54 PM UTC

I disagree with 1.3. "Serving static content is the easiest possible task for any web server." Yes, but keeping connections open for slow clients (esp with KeepAlive on) is not a good use of your 500MB Mongrel process' time. On the other hand, KeepAlive is a handy thing to have.

Using a proxy like nginx or varnish to serve static files (and even dynamic data) if you have the proper KeepAlive and Nagle bits flipped can save you a lot of server resources at the application layer.

Re: Sysadmin mistakes start-ups make

#7

Post by polvi » Tue, Oct 20, 2009, 5:55 PM UTC

patio11 wrote: post #4

If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application. I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning. Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently makin…

The solution to use an image processing library such as RMagick, http://rmagick.rubyforge.org/

Re: Sysadmin mistakes start-ups make

#8

Post by boundlessdreamz » Tue, Oct 20, 2009, 5:55 PM UTC

patio11 wrote: post #4

If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application. I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning. Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently makin…

Use a queue. You should never be doing time consuming method calls inside a controller anyway.

Re: Sysadmin mistakes start-ups make

#9

Post by peterwwillis » Tue, Oct 20, 2009, 5:56 PM UTC

The memory use is not accurate unless you take shared pages into account. Copy-on-write will make it look like each apache child is using 40MB, when really it's only 10MB private RSS. Use a RSS-calculating script (http://psydev.syw4e.info/new/misc/meminfo.pl) to determine the close-to-real memory use. If you don't calculate your maximum memory use correctly you will run into swap with traffic peaks. Also keep in mind that swap is a good thing. Is your app constantly cycling children? This isn't going to allow it to move unused/shared memory into swap. Don't ignore memory leaks by reducing your max requests per child.

The forking thing is more of the same. Copy-on-write means it's not going to balloon your memory unless some function turns that shared rss into private. It isn't something that you want to do a lot of, though.

Re: Sysadmin mistakes start-ups make

#10

Post by swombat » Tue, Oct 20, 2009, 5:58 PM UTC

patio11 wrote: post #4

If you fork inside an app server, such as mod_python, you will fork the entire parent process (apache!). This could happen by calling something like os.system("mv foo bar") from a python application. I nominate this post as the most distressingly important bit of information I've ever received at 2:43 AM in the morning. Now the question: what can I do in Ruby to avoid the four calls a second or so I'm currently makin…

I'd use something like DelayedJob and send_later the call to your image processing stuff, that way the forking happens out of the request path, at least.