AWS: the good, the bad and the ugly

#1

Post by brkcmd » Tue, Dec 18, 2012, 10:16 PM UTC

AWS: the good, the bad and the ugly
blog.awe.sm

Re: AWS: the good, the bad and the ugly

#2

Post by dirktheman » Tue, Dec 18, 2012, 10:49 PM UTC

Excellent post, and it reflects my experience with AWS, too. I really like the concept, and love S3 (and Glacier), EBS is just too unreliable for us to rely on for our entire business. Nice read, thanks!

Re: AWS: the good, the bad and the ugly

#3

Post by crcsmnky » Tue, Dec 18, 2012, 11:09 PM UTC

I'd be curious to hear about their backup/restore procedures with just ephemeral storage.

Re: AWS: the good, the bad and the ugly

#4

Post by FireBeyond » Tue, Dec 18, 2012, 11:11 PM UTC

Good article - it discusses things in a good way to get an overall quick look into how the ecosystem works.

I'm yet to play with AWS on any significant level, but this is the kind of thing to bookmark.

Further to that, are there any recommendations for books/sites/entries that discuss more best practices?

Re: AWS: the good, the bad and the ugly

#5

Post by IgorPartola » Tue, Dec 18, 2012, 11:12 PM UTC

> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime.

So what do you use now for your persistent storage? This might be the most interesting part.

Re: AWS: the good, the bad and the ugly

#6

Post by seldo » Tue, Dec 18, 2012, 11:15 PM UTC

crcsmnky wrote: post #3

I'd be curious to hear about their backup/restore procedures with just ephemeral storage.

We use Percona's XtraDB streaming backup to take backups of our smaller databases, and incremental backups of the larger DBs. We store them in a series of places: on a backup instance within east-1, on a second dedicated backup instance in west-1 (in case east-1 ever bites the dust completely, such as during hurricane Sandy), and then long-term archival on S3. S3 is good for smaller databases but for our biggest ones it takes multiple hours to download a full archive, hence the backups in multiple places.

However, our primary strategy for uptime is redundancy -- every db has at least one slave, and we are spread across multiple availability zones.

Re: AWS: the good, the bad and the ugly

#7

Post by seldo » Tue, Dec 18, 2012, 11:16 PM UTC

FireBeyond wrote: post #4

Good article - it discusses things in a good way to get an overall quick look into how the ecosystem works. I'm yet to play with AWS on any significant level, but this is the kind of thing to bookmark. Further to that, are there any recommendations for books/sites/entries that discuss more best practices?

I've not found any, which is part of why I wrote this post! It seems a lot of people blow time and money re-discovering these things.

Re: AWS: the good, the bad and the ugly

#8

Post by seldo » Tue, Dec 18, 2012, 11:16 PM UTC

IgorPartola wrote: post #5

> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime. So what do you use now for your persistent storage? This might be the most interesting part.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

Re: AWS: the good, the bad and the ugly

#9

Post by patrickgzill » Tue, Dec 18, 2012, 11:28 PM UTC

Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS?

Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn't hire someone.

Frankly, if your bills are greater than $3K per month with AWS I question whether you are truly saving anything.

(I figured that midsized instances vs. dedicated servers are about 5:1 in terms of performance)

Re: AWS: the good, the bad and the ugly

#10

Post by yegg » Tue, Dec 18, 2012, 11:28 PM UTC

seldo wrote: post #7

FireBeyond wrote: post #4

Good article - it discusses things in a good way to get an overall quick look into how the ecosystem works. I'm yet to play with AWS on any significant level, but this is the kind of thing to bookmark. Further to that, are there any recommendations for books/sites/entries that discuss more best practices?

I've not found any, which is part of why I wrote this post! It seems a lot of people blow time and money re-discovering these things.

http://alestic.com/ is the best thing I've come across (Ubuntu focused, but many general solutions as well).

FWIW, we (at DuckDuckGo) ended up in much the same place: ditched EBS, avoid anything that relies on EBS, and multi-zone and multi-region redundancy (also for latency purposes). For ephemeral storage purposes, we end up mainly using xlarge machines since they have the greatest stability and speed (with 4 drives in RAID-0).