Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS: the good, the bad and the ugly

blog.awe.sm

Post reply on HN
Print view View original

11–20 of 88 posts

Re: AWS: the good, the bad and the ugly

#11

Post by bps4484 »

seldo wrote: post #8
IgorPartola wrote: post #5

> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime. So what do you use now for your persistent storage? This might be the most interesting part.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about?

PS great article

Re: AWS: the good, the bad and the ugly

#12

Post by jeffbarr »

bps4484 wrote: post #11
seldo wrote: post #8

Earlier quoted context omitted.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about? PS great article

Each EC2 instance type has a different amount of ephemeral storage, from 160 GB to 3.3 TB. You need to arrange for it to be mounted when you launch the instance.

Re: AWS: the good, the bad and the ugly

#13

Post by stevencorona »

bps4484 wrote: post #11
seldo wrote: post #8

Earlier quoted context omitted.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about? PS great article

Even the smallest instances have 160GB of storage, all the way up to 3TB on the cc2.8xlarge. New high I/O instances have 2TB of "holy shit fast, destroys my $1000 enterprise SSD" storage. (just tested it last week)

Re: AWS: the good, the bad and the ugly

#15

Post by seldo »

bps4484 wrote: post #11
seldo wrote: post #8

Earlier quoted context omitted.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about? PS great article

As others have pointed out, all EC2 instances come with large ephemeral volumes already attached as /dev/sdb, etc; you just have to mount them:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/In...

Re: AWS: the good, the bad and the ugly

#16

Post by seldo »

patrickgzill wrote: post #9

Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS? Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn'…

If I knew how to setup OpenVZ and create Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month.

If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger, I think.

Re: AWS: the good, the bad and the ugly

#17

Post by dogas »

We here at PipelineDeals also abandoned EBS-backed instances after their 2nd outage.

Instead we rely on instances that use an instance-store root device. During the EBS outage, our instance store servers did not have any issues, while our EBS-backed servers really struggled throughout the day, with crazy high loads.

http://devblog.pipelinedeals.com/pipelinedeals-dev-blog/2012...

Re: AWS: the good, the bad and the ugly

#18

Post by joevandyk »

crcsmnky wrote: post #3

I'd be curious to hear about their backup/restore procedures with just ephemeral storage.

With postgresql, you can do streaming replication to a few machines. I have one large machine on standby, and then a couple really small ones, some on a different provider that just receive the updated db data using pg_receivexlog.

Stuff not in a database goes on S3.

Re: AWS: the good, the bad and the ugly

#19

Post by chimeracoder »

seldo wrote: post #8
IgorPartola wrote: post #5

> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime. So what do you use now for your persistent storage? This might be the most interesting part.

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

This is a good approach if you can get enough redundancy with your instances and if instance failure is uncorrelated.

However, what do you do if there's a massive outage that affects most or all of your instances simultaneously? Diversification that fails when you need it the most isn't very helpful.

Re: AWS: the good, the bad and the ugly

#20

Post by mey »

yegg wrote: post #10
seldo wrote: post #7

Earlier quoted context omitted.

I've not found any, which is part of why I wrote this post! It seems a lot of people blow time and money re-discovering these things.

http://alestic.com/ is the best thing I've come across (Ubuntu focused, but many general solutions as well). FWIW, we (at DuckDuckGo) ended up in much the same place: ditched EBS, avoid anything that relies on EBS, and multi-zone and multi-region redundancy (also for latency purposes). For ephemeral storage purposes, we end up mainly using xlarge machines since they have the greatest stability and speed (with 4 drive…

Is there not a feed for that site?
Post reply on HN