A newbie’s guide to scaling AWS

#1

Post by maxticket » Fri, Dec 30, 2011, 10:47 PM UTC

A newbie’s guide to scaling AWS
revisu.com

Re: A newbie’s guide to scaling AWS

#2

Post by christianreed » Fri, Dec 30, 2011, 11:11 PM UTC

Oh man, Fake Grimlock is the man! Er... Giant bird robot.

Re: A newbie’s guide to scaling AWS

#3

Post by njames029 » Fri, Dec 30, 2011, 11:12 PM UTC

Revisu is pretty sweet, great blog as well.

Re: A newbie’s guide to scaling AWS

#4

Post by mrmagooey » Sat, Dec 31, 2011, 12:07 AM UTC

I think I may have had less idea about what your product does after clicking "See Revisu in Action", three cat pictures does not a demo make.

Re: A newbie’s guide to scaling AWS

#5

Post by prib » Sat, Dec 31, 2011, 12:11 AM UTC

I can attest to the unreliability of AWS Micro instances for production environments. Two examples:

Ex 1: We had a request triggering an asynchronous job that would take 3-4 seconds of processor time. Several times, this job was being throttled, so even in a low traffic situation each time one of those requests came, a whole micro instance was consumed on it making it unavailable for anything else. Switching to small instances solved the problem.

Ex 2: When launching a new load balancer, due to some sort of wrong DNS/elastic-ip configuration, it's not uncommon to be hit hard by traffic coming for the last owner of the given ip. We've seen this happen several times in the past. Again, switching from 10+ micro instances to 2 small instances cleared the problem.

There's an interesting post by Greg Wilson addressing the throttling problem better than I could: http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instan...

Re: A newbie’s guide to scaling AWS

#6

Post by jaredstenquist » Sat, Dec 31, 2011, 12:21 AM UTC

Yes RDS, is great for the money considering the ease of backups and replicas, which normally are a pain in the ass for any sysadmin.

Overall this article has little to do with scaling on AWS. Running any sort of production website on a micro instance is pretty foolish, besides a blog or static website perhaps.

Re: A newbie’s guide to scaling AWS

#7

Post by blantonl » Sat, Dec 31, 2011, 12:22 AM UTC

As a large scale AWS user, I'm not sure I can agree with any of these recommendations.

Don’t use micro instances for real work

There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure.

Micro and small instances can be used for a personal Web site or tiny blog, but nothing else. When I visually think of a micro or small instance, I envision a PC sitting under a desk performing operations that a server in a rack should be hosting.

Have servers on standby. Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it.... ()

I really don't consider one server stopped, on standby, as scaling AWS. In the cases of Web servers, you might need to provision 5-10 Web servers, and having a single stopped instance is not going to allow you to accomplish that.

The better approach would be to have a server image prepared that can auto-provision itself (multiple times when needed) into the role defined through user-data passed to the image.

Scaling Web servers is an n+1 operation, not a +1 operation.

Seriously, use RDS

Seriously, if you are doing any real MySQL work on AWS, DO NOT use RDS. The reality is that AWS doesn't nearly do the EBS tuning that you yourself can do (RAID-0, kernel tuning etc) on your own EBS based MySQL implementation. See:

Getting Good IO from Amazon's EBS:

http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs...

Running MySQL on AWS:

http://aws.amazon.com/articles/1663

Finally, for MySQL slaves, use RAID-0 over the ephemeral disks which have dramatically better performance than EBS.

Re: A newbie’s guide to scaling AWS

#8

Post by maxticket » Sat, Dec 31, 2011, 12:28 AM UTC

mrmagooey wrote: post #4

I think I may have had less idea about what your product does after clicking "See Revisu in Action", three cat pictures does not a demo make.

Yeah, we're working on that. Hopefully the upcoming video helps a bit more. It's just one designer and one developer, so we gotta triage some things while we build the product.

Re: A newbie’s guide to scaling AWS

#9

Post by typicalrunt » Sat, Dec 31, 2011, 12:55 AM UTC

blantonl wrote: post #7

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or…

Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it.

I agree with allowing AWS to auto-provision N new instances when needed.

But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?

Re: A newbie’s guide to scaling AWS

#10

Post by chaz » Sat, Dec 31, 2011, 1:02 AM UTC

blantonl wrote: post #7

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or…

Good advice. But I think the context here is that there's a sudden spike of traffic from unexpected popularity that's likely to die out within a day or two. The mission is simply to survive that period.

If you're on Amazon EC2 or similar, it becomes phenomenally cheap to throw hardware at the problem. Running a single medium instance is $0.175/hour. If you need need to go from 5 servers to 50 servers for two days, it's just $420 to get out of a 48 hour jam. Need to upgrade to the beefiest 68 GB, 28 ECU RDS instance while your'e at it? $125 for 48 hours.

Performance tuning for 100x load that may never come again is usually hard to justify, so just know what your outs are if you're lucky enough to be swamped with traffic.

However, just be sure that you know how to scale out quickly and you can launch new instances (hopefully automatically). On most days, our eCommerce business gets by on just a couple of servers, but we have our autoscaling rules in place that will spin up/down servers within a few minutes.

Be sure to test regularly. Having a server boot up and add itself to the load balancer with old code/configs can wreak havoc on your database.