Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: the good, the bad and the ugly

blog.awe.sm

Post reply on HN
Print view View original

31–40 of 88 posts

Re: AWS: the good, the bad and the ugly

#31

Post by trotsky »

The elasticity, no capex and api management are great. But bending over backwards to deal with their terrible SAN, saturated layer 2 and low reliability easily wastes at least as much engineer time as it saves in ops. If the OP's 100 boxes are similar to most of the other AWS deployments I'm familiar with it's likely that'd be 10 or less actual machines to manage.

Don't get me wrong - the idea is great and it's great for really low end workloads where the gotchas don't matter or really big systems where you'd be managing against most of those issues anyway.

But in the middle of the two there is a dead space that I bet a lot of shops are stuck in - double digit instance numbers running a workload that a 2-4 servers could handle with plenty of leg room.

If you're deciding on a cloud provider in 2012 I think it makes a lot of sense to shop around. There are lots of people doing the on demand api deployment thing now with different trade offs. I like joyent a lot (local reliable io) or providers with cloud and a colo area even if it's exorbitant - as paying five hundred dollars a month for instnaces one ssd could replace sucks.

Re: AWS: the good, the bad and the ugly

#32

Post by patrickgzill »

seldo wrote: post #6
crcsmnky wrote: post #3

I'd be curious to hear about their backup/restore procedures with just ephemeral storage.

We use Percona's XtraDB streaming backup to take backups of our smaller databases, and incremental backups of the larger DBs. We store them in a series of places: on a backup instance within east-1, on a second dedicated backup instance in west-1 (in case east-1 ever bites the dust completely, such as during hurricane Sandy), and then long-term archival on S3. S3 is good for smaller databases but for our biggest ones…

did you ever examine DRDB (I think it is integrated into some of the Linux kernels)?

Re: AWS: the good, the bad and the ugly

#33

Post by cperciva »

jeffbarr wrote: post #12
bps4484 wrote: post #11

Earlier quoted context omitted.

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about? PS great article

Each EC2 instance type has a different amount of ephemeral storage, from 160 GB to 3.3 TB. You need to arrange for it to be mounted when you launch the instance.

a different amount of ephemeral storage, from 160 GB to 3.3 TB

Except for t1 and m3, which have zero ephemeral storage.

Re: AWS: the good, the bad and the ugly

#34

Post by dredmorbius »

jbeda wrote: post #25
dredmorbius wrote: post #23

Earlier quoted context omitted.

> If I knew how to setup OpenVZ and create I've got the Nobel Committee for Physics on line 2 should you accomplish that trick for DCs more than 30 km (18.6 mi) distant. Edit: Nevermind (misplaced decimal).

10 msec * c = ~3000 km or ~1860 miles. Of course light doesn't travel that fast through glass and there is latency at the hardware on either end. ;)

Units foulup. I'd used 0.10ms, not 10ms.

Re: AWS: the good, the bad and the ugly

#35

Post by seldo »

patrickgzill wrote: post #26
seldo wrote: post #16

Earlier quoted context omitted.

If I knew how to setup OpenVZ and create Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month. If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger,…

I am not trying to be sarcastic, I am just tired of some of the AWS fanboi's (not you) who act as if AWS solves all of your problems immediately for $50 a month. There are real costs no matter which way you go. PS, would be very surprised if you had even 45ms latency between AWS-east in Virginia and any of their facilities on the west coast...

I wasn't taking you as sarcastic, nor was I attempting snark in my reply: there are real costs either way, and it's our current judgement that learning how to do a virtualized, distributed stack would be more expensive than getting AWS to do it for us. (But, if I'm being honest, we haven't done any math)

And the < 10ms latency I'm talking about is between zones within us-east; latency to the west coast is a lot worse, but we only have emergency failover capacity in west.

Re: AWS: the good, the bad and the ugly

#36

Post by jbarham »

blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

> We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com.

FYI http://www.1000tb.com/ is the website for a landscaping company, not about bandwidth...

Re: AWS: the good, the bad and the ugly

#37

Post by vosper »

blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

What do you see as the benefits of using Elasticache vs your own memcached servers? It seems to me that memcached is so simple to configure and manage that paying a premium to have someone else do it isn't that great of a win.

We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers.

Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!

Re: AWS: the good, the bad and the ugly

#38

Post by brianhc »

jbarham wrote: post #36
blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

> We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. FYI http://www.1000tb.com/ is the website for a landscaping company, not about bandwidth...

try 100TB.com

Re: AWS: the good, the bad and the ugly

#39

Post by [deleted] »

blantonl wrote: post #24

This is a great writeup and is completely on target for realistic deployments on AWS. We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post: * I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forwar…

[deleted]

Re: AWS: the good, the bad and the ugly

#40

Post by jwilliams »

Nice read, but I wish they had included the location (and AZs) that are in use. I've used Oregon, California and Virginia with different results.

The comment around Ubuntu is interesting and I wish there was more detail there.

We use mdadm to run RAID across multiple EBSes. mdadm is great, but has a kink that it will boot to a recovery console if there the volume is "degraded" (i.e. any failure). This is even if the volume is still completely viable due to redundancy. This is obviously very bad, as you've got no way of accessing the console. It's an unfortunate way to completely hose an instance.

It's an easy one to miss, as you rarely test a boot process with a degraded volume. When it happens though - hurts a lot.

(If you'd like to check on this, make sure you have "BOOT_DEGRADED=yes" in /etc/initramfs-tools/conf.d/mdadm).

Post reply on HN