Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: the good, the bad and the ugly

blog.awe.sm

Post reply on HN
Print view View original

21–30 of 88 posts

Re: AWS: the good, the bad and the ugly

#21

Post by seldo »

dogas wrote: post #17

We here at PipelineDeals also abandoned EBS-backed instances after their 2nd outage. Instead we rely on instances that use an instance-store root device. During the EBS outage, our instance store servers did not have any issues, while our EBS-backed servers really struggled throughout the day, with crazy high loads. http://devblog.pipelinedeals.com/pipelinedeals-dev-blog/2012...

Great post! It's fascinating to see that other people have come up with the exact some solution -- we also run a "skeleton crew" server setup in us-west.

Re: AWS: the good, the bad and the ugly

#22

Post by seldo »

chimeracoder wrote: post #19
seldo wrote: post #8

Earlier quoted context omitted.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

This is a good approach if you can get enough redundancy with your instances and if instance failure is uncorrelated. However, what do you do if there's a massive outage that affects most or all of your instances simultaneously? Diversification that fails when you need it the most isn't very helpful.

As I mention in the post, we go across AWS availability zones: these systems have physically separate power and networking. When they fail together, it's because they are connected at the software level -- usually EBS. If you're not using EBS, then Amazon's multi-AZ pattern is sufficient: just make sure your instances are spread across AZs.

Of course, east-1 has fallen off the map entirely on at least one occasion -- for that reason we keep a "seed" set of warm databases in us-west; if east-1 were to have an extended outage we have a disaster recovery plan that involves transferring DNS to IPs in west and spinning up new app instances there.

Re: AWS: the good, the bad and the ugly

#23

Post by dredmorbius »

seldo wrote: post #16
patrickgzill wrote: post #9

Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS? Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn'…

If I knew how to setup OpenVZ and create Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month. If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger,…

> If I knew how to setup OpenVZ and create I've got the Nobel Committee for Physics on line 2 should you accomplish that trick for DCs more than 30 km (18.6 mi) distant.

Edit: Nevermind (misplaced decimal).

Re: AWS: the good, the bad and the ugly

#24

Post by blantonl »

This is a great writeup and is completely on target for realistic deployments on AWS.

We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post:

* I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens.

* We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only.

* If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs.

* RDS is still just a dream for us, since we still don't have a comfort level with performance.

* Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances.

* We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS.

* We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.

* If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.

Re: AWS: the good, the bad and the ugly

#25

Post by jbeda »

dredmorbius wrote: post #23
seldo wrote: post #16

Earlier quoted context omitted.

If I knew how to setup OpenVZ and create Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month. If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger,…

> If I knew how to setup OpenVZ and create I've got the Nobel Committee for Physics on line 2 should you accomplish that trick for DCs more than 30 km (18.6 mi) distant. Edit: Nevermind (misplaced decimal).

10 msec * c = ~3000 km or ~1860 miles.

Of course light doesn't travel that fast through glass and there is latency at the hardware on either end. ;)

Re: AWS: the good, the bad and the ugly

#26

Post by patrickgzill »

seldo wrote: post #16
patrickgzill wrote: post #9

Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS? Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn'…

If I knew how to setup OpenVZ and create Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month. If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger,…

I am not trying to be sarcastic, I am just tired of some of the AWS fanboi's (not you) who act as if AWS solves all of your problems immediately for $50 a month.

There are real costs no matter which way you go.

PS, would be very surprised if you had even 45ms latency between AWS-east in Virginia and any of their facilities on the west coast...

Re: AWS: the good, the bad and the ugly

#27

Post by pearkes »

blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

> We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.

Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?

Re: AWS: the good, the bad and the ugly

#28

Post by ceejayoz »

seldo wrote: post #6
crcsmnky wrote: post #3

I'd be curious to hear about their backup/restore procedures with just ephemeral storage.

We use Percona's XtraDB streaming backup to take backups of our smaller databases, and incremental backups of the larger DBs. We store them in a series of places: on a backup instance within east-1, on a second dedicated backup instance in west-1 (in case east-1 ever bites the dust completely, such as during hurricane Sandy), and then long-term archival on S3. S3 is good for smaller databases but for our biggest ones…

Any good resources for XtraDB's streaming backups? I'm looking to move in that direction early next year.

Re: AWS: the good, the bad and the ugly

#29

Post by thisishugo »

mey wrote: post #20
yegg wrote: post #10

Earlier quoted context omitted.

http://alestic.com/ is the best thing I've come across (Ubuntu focused, but many general solutions as well). FWIW, we (at DuckDuckGo) ended up in much the same place: ditched EBS, avoid anything that relies on EBS, and multi-zone and multi-region redundancy (also for latency purposes). For ephemeral storage purposes, we end up mainly using xlarge machines since they have the greatest stability and speed (with 4 drive…

Is there not a feed for that site?

http://feeds.alestic.com/alestic

Re: AWS: the good, the bad and the ugly

#30

Post by seldo »

blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

My ops guy has reminded me that we do, like you, have one EBS drive for snapshots of one particularly hefty DB in addition to our other backup methods.
Post reply on HN