Holy crap. An Amazon rep actually just posted that SkyNet had nothing to do with the outage:
https://forums.aws.amazon.com/message.jspa?messageID=238872#...
Amazon Web Services are down
271–280 of 346 posts
Re: Amazon Web Services are down
#272Earlier quoted context omitted.
There are subscriptions for various levels of support, from $50/mo for 12 hour response time to $15k/mo for 15 minute response time. http://aws.amazon.com/premiumsupport/
Interesting pricing. The Platinum seems priced to have no one use it, considering how much of a jump it is over Gold.
I would say, rather, that it is priced to have very specific sorts of customer using it.
The relationship between the pricing tiers changes fairly drastically, depending on how much you are already spending on Amazon Web Services. Gold, for instance, starts out at 4x the price of Silver support, but by the time you're spending 80K/month on services, it's only a $900 premium (and stays there no matter how much bigger your bill is). At the $150K/month level, it's a 2x jump from Gold to Platinum, which may or may not be a huge jump, considering the extra level of service you get.
Re: Amazon Web Services are down
#273Earlier quoted context omitted.
"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
It's a strange algebra though; doesn't it mean the WORSE Amazon's uptime is, the more money you should give them?
More accurately, the more unstable your infrastructure is, the more you will need to spend to ensure stability.
Re: Amazon Web Services are down
#274Earlier quoted context omitted.
You guys might have answered this in one of your AMAs/blog posts (or was it raldi who commented?), but what options can reddit resort to should this stuff happen again to this degree of severity?
We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.
How do you move away from EBS and still deal with large data?
Re: Amazon Web Services are down
#275Today, April 21st 2011, according to the "Terminator", Skynet was launched... No wonder AWS is down
Re: Amazon Web Services are down
#276I guess this is one Reddit outage that can't be blamed on poor scaling
Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.
Sup jedberg, I obviously don't have nearly the level of knowledge with the intricacies of reddit, but coming from a strictly "business" standpoint, the amount of downtime reddit receives due to amazon issues is astounding. Perhaps it's time to look for alternatives?
Anyway, thanks for your time.
Re: Amazon Web Services are down
#277Earlier quoted context omitted.
A couple of years ago you had expressed interest in making a port to App Engine, any interest in doing that still? Want any help? ;)
I think it would take a lot more time than we have to make that work. Our code is open source if you want to give a proof of concept a go. ;)
On the off chance a port to app engine coalesces around this comment, count me in :)
Re: Amazon Web Services are down
#278Re: Amazon Web Services are down
#279Earlier quoted context omitted.
Reddit's been down for several hours today, I'm sure they are already way lower 99.95%.
http://cache-scale.appspot.com/c/www.reddit.com/ That's a cache of it. I really wish that the admins at reddit would implement something like this themselves, then link to it when downtime like this happens.
do you have any stats of this app in terms of total data stored, bandwidth required per day, requests per second?
Re: Amazon Web Services are down
#280Some quotes regarding how Netflix handled this without interruptions: "Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1] "Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2] [1] https://twitter.com/adrianco/status/61075904847282177 [2] https://twitter.com/adrianco/status/61076362680745984
"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
Every decision in a business is like this - measure the cost of action A versus the cost of not-A. It's just rare that in this case, those costs are easily quantifiable.