Live data from Hacker News

A newbie’s guide to scaling AWS

revisu.com

11–20 of 21 posts

Re: A newbie’s guide to scaling AWS

#11
post #7

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or…

Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it. I agree with allowing AWS to auto-provision N new instances when needed. But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?

You do not pay for an EC2 instance that is stopped. If the instance has attached EBS volumes you will continue to pay for the storage costs, but that is pretty minimal.

It is important to remember that a stopped instanced is not guaranteed to be able to be started again. If you have a m1.large stopped, there is a chance that Amazon will not have any m1.larges available in that zone/region when you try to start back up your instance.

Also know that the private dns name will change when a stopped instance is started again, so anything configured to point at that particular instance will have to be reconfigured.

Re: A newbie’s guide to scaling AWS

#12
post #7

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or…

Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it. I agree with allowing AWS to auto-provision N new instances when needed. But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?

But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?

You aren't paying for the CPU time, only the S3/EBS storage space that the image occupies (which is negligible for any normal single instance/image)

Re: A newbie’s guide to scaling AWS

#14
post #7

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or…

Configure a large instance, stick it in an Elastic Load Balancer with your other web servers, and just stop it. I agree with allowing AWS to auto-provision N new instances when needed. But from the article and with my financial hat on, when the large instance is stopped and left in a standby role, are you still paying for it?

[deleted]

Re: A newbie’s guide to scaling AWS

#16
I've been using a Micro Instance for almost a year (the free tier) to run http://nsfw.in/ successfully. And the traffic on the site is also not bad at all, with 100,000+ pageviews a month on an average. You've to know the right thing you need for your requirement and tweak it.

Re: A newbie’s guide to scaling AWS

#17
post #7

As a large scale AWS user, I'm not sure I can agree with any of these recommendations. Don’t use micro instances for real work There are some use cases where micro instances work in production environments (such as MongoDb arbitrators etc) but running anything in production on AWS on anything less than a m1.large instance is setting you up for failure. Micro and small instances can be used for a personal Web site or…

Agreed on all the scaling stuff - if you really serve traffic, m1.large should be your go-to instance size unless you're sure you know better, and being "ready to scale" means that if you need to serve 2x, 20x, or 100x the traffic, you can have 2, 20, or 100 servers booting up and self-configuring (including adding themselves to the load balancer) at the click of a button. Not that you have a few servers sitting around on standby waiting to be enabled...

There's one exception: if you've got "internal" servers that are only serving requests from you, or running occasional cron jobs (for monitoring and stuff like that), and you're sure that user traffic changes will never affect them, by all means go for a micro or a small. Those things aren't part of your publicly visible stack, so there's no sense wasting money.

Seriously, if you are doing any real MySQL work on AWS, DO NOT use RDS. The reality is that AWS doesn't nearly do the EBS tuning that you yourself can do (RAID-0, kernel tuning etc) on your own EBS based MySQL implementation.

Do you have any numbers on the performance difference between a well tuned EBS-based implementation and RDS?

Because speaking from experience, EBS based MySQL machines are always, without fail, the primary maintenance headache in any stack. They're always the machines that wake engineers up at 4 AM because of some EBS crappery or somesuch nonsense that necessitates intervention, and the promotion/backup/restore procedures can be touchy and error-prone. That's not even to mention the work involved in setting them up, testing, and getting them tuned right, which are serious stuff compared to the "pick a name, size, version, and launch!" process that RDS provides.

Most of what I could find seemed to suggest that the performance differences between RDS and well tuned EBS MySQL servers of comparable size were relatively small. For instance, from http://www.mysqlperformanceblog.com/2011/08/04/mysql-perform...: "My benchmarks generally showed that I was not able to outstrip RDS 5.1′s performance with the combination of stock MySQL 5.1, EC2, and a 10-volume EBS RAID."

If you think that's wrong and the difference is really much higher, I'd be interested to know.

Part of me is skeptical that any serious differences would persist for very long: if it's so easy to do this "right" so that you end up with a EBS+EC2 MySQL server that significantly outperforms RDS for general-purpose workloads, why wouldn't the RDS team set up RDS servers that way behind the scenes? I would think that Amazon should know how to efficiently utilize its own EBS volumes, right?

Re: A newbie’s guide to scaling AWS

#18
One of our problems in scaling is not starting new instances, but rather to ensure that new instances have the latest version of the platform.

Also, we are running a fantasy soccer game with trading windows, so the server load near the trading window closure time (the last hour or so) is orders of magnitude higher than average load. Ideally we would scale up when the load demands it, but with the platform versioning issue unresolved we need to run our platform at "full scale" almost constantly (or scale manually which is a really bad idea when someone forgets to scale back up again for the weekend).

We're running in a WISC setup.

Re: A newbie’s guide to scaling AWS

#20

One of our problems in scaling is not starting new instances, but rather to ensure that new instances have the latest version of the platform. Also, we are running a fantasy soccer game with trading windows, so the server load near the trading window closure time (the last hour or so) is orders of magnitude higher than average load. Ideally we would scale up when the load demands it, but with the platform versioning…

What if every time you did a deploy, afterwards you kicked off a job that wrote a snapshot and brought up instances based off that. Then at any point you'd only ever be at most a deploy behind.
Post reply on HN