Live data from Hacker News

AWS: the good, the bad and the ugly

blog.awe.sm

51–60 of 88 posts

Re: AWS: the good, the bad and the ugly

#51

Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS? Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn'…

You can find essays from AWS users who swear it would be prohibitive--financially and mentally--to ever switch.

And you can find essays of people who run their own rack who swear the same thing about moving to AWS.

I worked at an all-AWS bay area startup that ran a $100k/mo Amazon bill, and now I work at a bay area startup that runs all its own hardware.

It's about tradeoffs. It's not just about cost. (Though there are many cases where Amazon is cheaper and it has little to do with your monthly spend, and certainly not an arbitrary threshold like $3k/mo.)

Writing this at the close of my employer's Open Enrollment period, I'd say this: Comparing AWS with running your own hardware is like trying to compare two health insurance plans. Each brings its own seemingly impossible tradeoffs. All you can do is bring your experience and judgment to identify core issues, make the best call you can, and suppress the urge to dream of the road not taken when you're knee deep in whatever hell you're experiencing that would be a non-issue if only you had went the other way.

Though if you loathe the AWS "fanboyism" so much then maybe it's the company you're keeping: Few currencies in a startup are as valuable as flexibility. The younger the startup, the truer that is.

Re: AWS: the good, the bad and the ugly

#52

Earlier quoted context omitted.

I am not trying to be sarcastic, I am just tired of some of the AWS fanboi's (not you) who act as if AWS solves all of your problems immediately for $50 a month. There are real costs no matter which way you go. PS, would be very surprised if you had even 45ms latency between AWS-east in Virginia and any of their facilities on the west coast...

Much of the attraction that I see from people comes from the flexibility. Once you get to any significant size in an organization, it's not unusual for people to have to wait for _months_ to get servers rolled out. The ability to whip out your credit card, and fire up new servers at AWS in minutes, becomes very attractive in that scenario.

Ghshephard nailed it, as usual.

Re: AWS: the good, the bad and the ugly

#53

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

I don't get the 100tb.com model. Checking out their website, I see: Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server. 250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month. So either: A) Their connectivity isn't hot. B) If yo…

Something that you want to keep in mind with 100TB is that they have pulled companies before, and left them in pretty dire straights.

http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale...

I don't want to try to guess why they did that, but it's something that should never be forgotten.

Re: AWS: the good, the bad and the ugly

#54

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

I don't get the 100tb.com model. Checking out their website, I see: Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server. 250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month. So either: A) Their connectivity isn't hot. B) If yo…

Something that you want to keep in mind with 100TB is that they have pulled companies before, and left them in pretty dire straights.

http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale...

I don't want to try to guess why they did that, but it's something that should never be forgotten.

Re: AWS: the good, the bad and the ugly

#55
post #47

Earlier quoted context omitted.

Even the smallest instances have 160GB of storage, all the way up to 3TB on the cc2.8xlarge. New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage. (just tested it last week)

New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage. Just a tongue-in-cheek comment: Those instances also cost over $2000 USD per month. For the same money you could buy a new Dell Server with 2-4 SSDs, every month. ;)

THey're only $2000 if you don't have reserved instances.

Re: AWS: the good, the bad and the ugly

#56

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

> RDS is still just a dream for us

In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over.

RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.

Re: AWS: the good, the bad and the ugly

#57
post #53

Earlier quoted context omitted.

I don't get the 100tb.com model. Checking out their website, I see: Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server. 250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month. So either: A) Their connectivity isn't hot. B) If yo…

Something that you want to keep in mind with 100TB is that they have pulled companies before, and left them in pretty dire straights. http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale... I don't want to try to guess why they did that, but it's something that should never be forgotten.

So - that pretty much answers my question. If you actually start to use that 100TB, they drop you as a customer. TANSSAAFL, so, if it looks to good to be true ($0.33/mbit/second on Softlayer network) - then it probably is.

I'd be very interested though, in what people have been able to sustain per server at 100TB without getting dropped.

My guess is that it's somewhere around 32 Terabytes/month per server before 100TB gets a little angsty (100 mbits/second sustained), particularly if you have more than a dozen or so servers with them (filling out a GigE sustained)

Re: AWS: the good, the bad and the ugly

#58
From alestic.com

>> Both EBS boot and instance-store AMI ids are listed, but I recommend you start with EBS boot AMIs.

Why two opposite recommendations from alestic.com [authority on AWS] and practitioners?

Not a flame - I am planning my AWS deploy strategy and need to make a decision between these two approaches.

Re: AWS: the good, the bad and the ugly

#60
post #17

We here at PipelineDeals also abandoned EBS-backed instances after their 2nd outage. Instead we rely on instances that use an instance-store root device. During the EBS outage, our instance store servers did not have any issues, while our EBS-backed servers really struggled throughout the day, with crazy high loads. http://devblog.pipelinedeals.com/pipelinedeals-dev-blog/2012...

Really interested to learn about how you use Chef to bring up and configure your ephemeral instances.

Also, do you backup your databases anywhere other than on other ephemeral instances?

Post reply on HN