291–300 of 346 posts
I guess this is one Reddit outage that can't be blamed on poor scaling
Thankfully, no. :) But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.
http://cache-scale.appspot.com/c/www.reddit.com/
but officially supported (and paid for) by you?
Earlier quoted context omitted.
Please do not make content-free posts such as this. It adds no value to the conversation and is only noise. If you actually wish to make a useful point about the practicality or otherwise of massively virtualised systems for webapp deployment, please do. It's going to take more than two words though.
I guess you haven't seen the Microsoft ads about the cloud? http://www.youtube.com/watch?v=Lel3swo4RMc Anyway, my bad, I was just trying to make a joke to lighten up the mood. Sorry.
Some quotes regarding how Netflix handled this without interruptions: "Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1] "Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2] [1] https://twitter.com/adrianco/status/61075904847282177 [2] https://twitter.com/adrianco/status/61076362680745984
"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
Earlier quoted context omitted.
"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
It's a strange algebra though; doesn't it mean the WORSE Amazon's uptime is, the more money you should give them?
Earlier quoted context omitted.
"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.
I have not had a single service issue with them, ever. They do a better job at reliably providing me with TV shows than the cable company does. That seems to be where they're looking to position themselves, and the reputation for always being there is hard to regain if you lose it.
Earlier quoted context omitted.
We're moving away from the EBS product altogether. The hard part is dealing with the master databases. Normally I'd have a master database with a built in raid-10, but I can't do that on EC2, so I have to come up with another option. So I guess that is the long way of saying that hopefully it won't happen again.
How do you move away from EBS and still deal with large data?
1. Sharding data 2. Pulling tables out to other servers from the main DB 3. Pruning excessive data 4. Compressing data
Earlier quoted context omitted.
Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one regio…
No, your decimal point is off. 0.05% * 0.05% = 0.0005 * 0.0005 = 0.00000025, or 0.000025%. It works out to an expected downtime of 8 seconds per year, instead of over 4 hours for one location. Of course, redundancy doesn't set itself up, so there are added costs on top of Amazon.
Earlier quoted context omitted.
Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region. Good point. let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one regio…
Why wouldn't a simple expected value calculation work? You've shown that you can calculate the extra availability that subscribing to another region provides. Simply multiply the cost of an outage by the extra availability provided by an additional region that would have prevented that outage. If expanding to another region costs more than just taking the outage, then it's categorically not a good option. If manageme…
Earlier quoted context omitted.
"Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.
I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.
This is even more true if/when you load balance between zones and aren't just using them as hot backups. As another commenter pointed out, Netflix says they have three zones and only need two to operate.