Silver lining: Hopefully I can test my "aws is failing" fallback code. (my GAE based site keeps a state log on S3 for the day when GAE falls in a hole.)
Amazon Web Services are down
151–160 of 346 posts
Re: Amazon Web Services are down
#152Earlier quoted context omitted.
If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.
The whole point of AWS is to forget about maintaining hardware infrastructure. Amazon are the ones who should have made backups in multiple regions, and transfer the load on failure.
Actually, the whole point of AWS is to have options for using hardware that you don't own. They don't offer any magic "all your stuff in one package, guaranteed to work all the time" service. So yes, you do still need to think about your hardware infrastructure. You just don't have to own it.
And Amazon does have all their stuff available in multiple regions. It's up to you to use it though.
Re: Amazon Web Services are down
#153Earlier quoted context omitted.
Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.
0.05% of one year is 4 hours, 22 minutes and 48 seconds.
Reddit experienced some issues with Amazon a month ago that resulted in the site being down for almost a day. I'm pretty sure they're way below that percentage.
Re: Amazon Web Services are down
#154Earlier quoted context omitted.
Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.
If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.
He wasn't suggesting that all Reddit's problems are due to Amazon services, he was using Reddit's down time today as a data point illustrating that the uptime guarantee claimed for the service has not been kept this year (in fact a whole year's "permitted downtime" as implied by the 99.95% SLA may be eaten on one day). Presumably Amazon will be handing out some refunds and other compensation (assuming the SLA isn't of the toothless "it'll be up at least 99.95% of the time, unless it isn't" variety).
Perhaps the Reddit admins decided that "up to 0.05%" downtime permitted by the SLA would be acceptable, compared to the extra expense of using more of Amazon's services (and any coding/testing time they may have needed to take advantage of the redundancy depending on how automatic the load balancing and/or failover are within Amazon's system) to improve their redundancy. By my understanding the promise isn't 99.95% if you use more than one of our locations, it is 99.95% at any one location, so the fact that Reddit don't make use of more than one location is irrelevant when talking about the one location they do use not meeting the expectations listed in the SLA.
I'm not saying Reddit's implementation decision is right (I don't have the metrics available to make such a judgement) but it would have been made based partly on that 99.95% figure and how much they trusted Amazon's method of coming to that figure as a reliability they could guarantee. If I had paid money for a service with a 99.95% SLA, unless the SLA had no teeth, I would be expecting some redress at this point (though there is probably no use nagging Amazon about that right now: let them concentrate on fixing the problem and worry about explanations/blame/compo later once things are running again).
Re: Amazon Web Services are down
#155Re: Amazon Web Services are down
#156Re: Amazon Web Services are down
#157Earlier quoted context omitted.
Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.
If the Reddit web server admins took availability seriously they would have chosen to deploy across more than one region. Do you disagree? Why do you disagree? I'm being honest, no snark involved in my questions.
> If the Reddit web server admins took availability seriously
We do.
> they would have chosen to deploy across more than one region.
It's far too costly to do that. We are deployed across multiple AZs, but this failure hit multiple AZs.
Re: Amazon Web Services are down
#158Re: Amazon Web Services are down
#159Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode. The increased complexity of their offering makes it more likely that things will break, leaving you locked in. I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.u…
Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?" Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime…
I agree with your sentiments, but this:
> I always respond: "And why do you believe that you would do any better?"
…is an apples-to-oranges comparison that implies that the thing that Amazon is currently doing is what you would be doing in an Amazon-less scenario. It's not.
Amazon hosts thousands of customers and needs to service all of them in the same infrastructure, which coincidentally is also shared between every customer by virtue of being virtualized. It's a complex structure in which every resource -- CPU, RAM, disk, network -- is shared, which means that even a minor EBS problem could potentially have a butterfly-effect propagating through multiple customer's hosts. There are limitations imposed by the structure; for example, the only way to access large amounts of block storage with local-like performance is through EBS. But EBS has been shown to be high-latency and flaky, and Reddit's chronic problems with EBS is a good example why it one probably should stay away from it entirely. Network latency on EC2 is also pretty horrendous compared to classic non-virtual setups; I have set up HAProxy clusters where the connection setup time to backend hosts would be measured in tens of milliseconds.
Now, if you hosted your own stuff, you would be the only customer, and you would not be sharing resources with anyone else, and you could design your hardware exactly to your specifications (fast local disks with low latency, fast networking with low latency and so on). The difference in complexity is significant. There are tons of challenges and costs involved in hosting your own stuff and maintaining uptime, but the complexity equation is different.
Re: Amazon Web Services are down
#160Earlier quoted context omitted.
Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?" Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime…
The problem with Amazon is that despite touting an open API, their infrastructure internals and practices are a trade secret, so the likes of Eucalyptus are having to play catchup. In other words, I cannot replicate their infrastructure in my own data center, even I had the money to pay them. I suspect this is the main reason that Heroku didn't move off Amazon, and not the fact that Amazon was providing them great va…
That's where I think we have to give credit to Rackspace for open sourcing all (or much) of their tech behind Rackspace Cloud