Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Post-Mortem

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: AWS Post-Mortem

#51

Post by wes-exp »

Locke1689 wrote: post #39
wes-exp wrote: post #31

Earlier quoted context omitted.

Only fools would want a persistent filesystem for their OS!

How is S3 not persistent...

Notice I said for the OS. From Wikipedia:

"An EC2 instance may be launched with a choice of two types of storage for its boot disk or "root device". The first option is a local "instance-store" disk as a root device (originally the only choice). The second option is to use an EBS volume as a root device.

Instance-store volumes are temporary storage, which survive rebooting an EC2 instance, but when the instance is terminated (e.g., by an API call, or due to a failure), this store is lost." (Emphasis mine)

Most choose the EBS type of instance now, because having the OS on a persistent storage device is more convenient for typical server scenarios (as you might imagine).

http://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud#Pe...

Re: AWS Post-Mortem

#52

Post by Locke1689 »

wes-exp wrote: post #51
Locke1689 wrote: post #39

Earlier quoted context omitted.

How is S3 not persistent...

Notice I said for the OS . From Wikipedia: "An EC2 instance may be launched with a choice of two types of storage for its boot disk or "root device". The first option is a local "instance-store" disk as a root device (originally the only choice). The second option is to use an EBS volume as a root device. Instance-store volumes are temporary storage , which survive rebooting an EC2 instance, but when the instance is…

The only scenario I can see that would require hitting an EBS store often enough to matter would be if you put your database on it. Obviously your OS kernel is going to be in resident memory, so your argument is somewhat of a strawman.

Re: AWS Post-Mortem

#53

Post by lmz »

Locke1689 wrote: post #52
wes-exp wrote: post #51

Earlier quoted context omitted.

Notice I said for the OS . From Wikipedia: "An EC2 instance may be launched with a choice of two types of storage for its boot disk or "root device". The first option is a local "instance-store" disk as a root device (originally the only choice). The second option is to use an EBS volume as a root device. Instance-store volumes are temporary storage , which survive rebooting an EC2 instance, but when the instance is…

The only scenario I can see that would require hitting an EBS store often enough to matter would be if you put your database on it. Obviously your OS kernel is going to be in resident memory, so your argument is somewhat of a strawman.

It's the root device that's on EBS. On most AMIs with EBS root that's the entire OS i.e., /etc, /dev, /bin, /usr, etc. and not just the OS kernel.

Re: AWS Post-Mortem

#55

Post by Locke1689 »

lmz wrote: post #53
Locke1689 wrote: post #52

Earlier quoted context omitted.

The only scenario I can see that would require hitting an EBS store often enough to matter would be if you put your database on it. Obviously your OS kernel is going to be in resident memory, so your argument is somewhat of a strawman.

It's the root device that's on EBS. On most AMIs with EBS root that's the entire OS i.e., /etc, /dev, /bin, /usr, etc. and not just the OS kernel.

And for the things that matter (i.e., not fake mount points) that will all mostly live in resident memory.

Re: AWS Post-Mortem

#57

Post by aaronbrethorst »

rapind wrote: post #33
saturn wrote: post #6

As someone who has put a considerable amount of resources moving things into cloud computing - I wanted to believe. But I have changed my mind. Cloud computing scales the efficiencies, yes. It also scales the problems. And because of this, AWS is by several orders of magnitude the worst of my current hosts. I have dedicated servers. No downtime in past year. I have a couple of cloud servers with rackspace. No downtim…

I'm the same. I fully embraced S3 and EC2 when they came out (even played with SQS) and enthusiastically told everyone I could that this was the future, it's the new electricity, etc. While I still think that eventually it will end up as a utility I'm opting out of the cloud for anything production for the time being. I'll keep an eye on it of course. Does anyone know if Heroku has spread their services across HA zon…

    Does anyone know if Heroku has spread their services across HA zones?
I don't know for a fact, but the multiple Heroku sites of mine that went down concurrently with this problem would indicate 'no.'

Re: AWS Post-Mortem

#58

Post by gfodor »

For all those complaining about AWS I think it's important to not fall into the trap of throwing all of Amazon's services into the same bucket. EBS (and hence, RDS) have shown time and time again to be the most complex offerings and more prone to failure.

Generally speaking, at least for now, the parts of your system built on top of EBS should be carefully architected to survive in the face of erratic EBS latency, data corruption, or even downtime. (All of which are part of the standard AWS contract, but happen much more often in practice than if you are used to the mean failure time of a hard disk sitting in a cage.)

This pattern leads me to believe that services such as VoltDB that do not directly rely upon attached storage will prove to be the paradigm necessary to get reliable cloud computing, at least in the AWS ecosystem. On-demand provisioning of disk is an extraordinary hard problem, and a world where local ephemeral storage provides durability through redundancy across nodes and AZ's is probably where we are headed.

Re: AWS Post-Mortem

#59

Post by grourk »

mtkd wrote: post #3

It's a good communication from Amazon - maybe a little too long - could use a summary block at top. The compensation looks generous too.

There's a summary block at the bottom -- but it's not a summary.

Re: AWS Post-Mortem

#60

Post by streptomycin »

tednaleid wrote: post #32
saturn wrote: post #21

Earlier quoted context omitted.

You mean availability zone. Well I guess me, Reddit, Foursquare, and plenty of other sites just got lucky in the bad availability zone. Ah yes, here's the classic AWS apologist pattern in full effect. You don't use EBS! Of course you don't, you would have to be some kind of friggin' idiot to use EBS. So what do you use for, say, MongoDB data files, that is different from morons like me who stupidly assumed they could…

We use EBS, and had machines in the availability zone that went down that were affected. Those machines were out for longer than a day, but we were back up within an hour because we had redundancies built in across other availability zones. If you're doing anything that matters, you can't rely on a single zone/machine/whatever, no matter who your hosting provider is.

So you're saying we need a cloud of clouds?
Post reply on HN