Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix is Down

gigaom.com

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Netflix is Down

#41

Post by jrockway »

toomuchtodo wrote: post #29

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering resources to fix the software.

(Why doesn't Amazon offer transparent replication? Because the price of replication is unbelievably high and most people can't believe how high it is. If you want to write to 12 replicas around the world, budget five seconds for your transaction to complete. Compare this to a MySQL database on an SSD that can do millions of writes on the same entity group in the same amount of time.

This doesn't even include the cost: 12x the storage cost, and 12x the bandwidth cost for data going from your frontend to all of the backends.)

Re: Netflix is Down

#42

Post by moe »

jrockway wrote: post #41
toomuchtodo wrote: post #29

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering…

You're blowing this a little out of proportion.

Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix.

You rarely hear of Google outages, or iTunes, or Youtube, or [insert six dozen other popular brands], do you? Yes it can happen to the best of them, but the EC2 outages are really piling up lately.

Re: Netflix is Down

#43

Post by toomuchtodo »

jrockway wrote: post #41
toomuchtodo wrote: post #29

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering…

Netflix: Spitting out 1 of 120 different encoded version of a video to a client device after its been authenticated.

So, you've got an authentication layer, which can pass a token to the browser or physical hardware client access device. This should be very lightweight, no? Even with millions of users, this database shouldn't be enormous.

And then you've got your encoded libraries. This should be data being spit off platers on CDN networks. CDNs aren't hard anymore. Cloudfront is cheap, or you can go with Akamai, Limelight, Level3, etc. Properly built, you should never NOT be able to serve from somewhere in the CDN, even if the serving location isn't optimal.

Yes, this is HEAVILY simplified. I've left out recommendations, their encoding process/infrastructure, etc. This is not atomic data that is being replicated in realtime; these are videos that are lazily encoded, stored, and then served on demand.

It should not be this hard for this use case.

Re: Netflix is Down

#44

Post by superuser2 »

toomuchtodo wrote: post #29

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

Because systems only crash when they're maintained by other people?

Re: Netflix is Down

#45

Post by jimwalsh »

gurch101 wrote: post #37

Having been in a similar position in the past, I actually feel bad for all the people at amazon and netflix that'll need to work late tonight...

They are getting OT/Holiday pay I'm sure. So they will be ok.

Re: Netflix is Down

#47

Post by 1qaz2wsx3edc »

jrockway wrote: post #41
toomuchtodo wrote: post #29

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering…

Just poking in on this, does anyone have a graph of outages over time? I'm wondering if it has been increasing or decreasing; and therefore if amazon is getting better at managing it's demand. Bonus points if it compares the size of the customer base over time in comparison or relation.

Re: Netflix is Down

#48

Post by moe »

muyuu wrote: post #38
zrail wrote: post #36

Earlier quoted context omitted.

Amazon EC2 has a Service Level Agreement (SLA)[1] that guarantees 99.95% uptime in a rolling 365 day period, and provides for service credits. Of course, a huge customer like Netflix can and probably did negotiate their own SLA terms. [1]: http://aws.amazon.com/ec2-sla/

I was wondering if they'd pay damages and how would that be calculated. I don't think free service for a while would cut it with a contingency of this sort. Maybe someone has first-hand information of their contract (or any other big player) and can answer to this publicly.

I don't know the Netflix contract but I've been involved with a few big (telco) SLAs. The penalties are usually calculated with a points-system and a multiplier that raises according to the duration/impact of an outage.

E.g. the first 15 minutes of an outage may cost 1 point per minute, 15-60 minutes 2 points, and so on. You also have multipliers for the severity (partial or full outage, customer impact, affected countries, etc.), time-of-day, and so on.

Collected points may then later be traded in for dollars or a nice lawsuit.

Corporate lawyers love to go nuts on these things, an enterprise SLA can easily span a hundred pages of legalese.

Re: Netflix is Down

#49

Post by toomuchtodo »

superuser2 wrote: post #44
toomuchtodo wrote: post #29

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

Because systems only crash when they're maintained by other people?

Assumption: Your business uses Amazon's EBS (because heh, you need to store your data somewhere between EC2 instance reboots/creation/destruction). EBS is down. What do you do? Hope AWS engineers get it back up fast? You can't do a damn thing about its reliability; you're stuck with whatever reliability Amazon has decided to deliver (which I think we can agree is much less than a company with revenue can depend on).

When you build your own system, you decide how reliable you want it. Otherwise, you can use Amazon and just live with the continual problems.

Post reply on HN