Live data from Hacker News

Netflix is Down

gigaom.com

71–80 of 81 posts

Re: Netflix is Down

#71
post #42

Earlier quoted context omitted.

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering…

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

You rarely hear of Google outages, or iTunes, or Youtube, or [insert six dozen other popular brands], do you? Yes it can happen to the best of them, but the EC2 outages are really piling up lately.

To me, this makes perfect sense. Google products are almost always designed to tolerate data center failures, and so you don't hear about data center failures affecting Google products. On the other hand, Joe's Random EC2 app is not designed for the same high availability, and so it's down whenever Amazon is. (This, incidentally, is a reasonable trade-off for most people. A few hours of downtime a year is often much cheaper than paying engineers to ensure that those few hours are five minutes or less.)

So the flaw here is not Amazon. The flaw is relying on one cluster of computers to handle all your computing needs. Amazon could make their product high-replication by default, but it would be so slow and expensive that nobody would use it. So the task is kicked to the application developers instead of the platform developers, and you should consider holding them responsible for the downtime.

(Incidentally, you can compare EC2 to AppEngine here. How often is your favorite AppEngine app down? Less than your favorite EC2 app, probably, because AppEngine pretty much forces the high-replication datastore, even though the semantics are strange for developers used to traditional single-homed or master/slave architectures. And if you read the AppEngine discussion groups, you'll see that users raise most of the concerns that I describe here: "Why is AppEngine more expensive than EC2?" "Why is AppEgnine slower than EC2?", and so on.)

Now, why does Netflix's video serving go down when an EC2 cell does? I have absolutely no idea. I do know that they do more than just stream bytes, though; they have to authenticate users, track what they're watching, and apply DRM to the streams. So there's that.

If you want fault-tolerant video distribution, look no further than the Pirate Bay. Although their servers are frequently seized, the video bits keep flowing. That's by design, not by accident.

Re: Netflix is Down

#73

You'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all

How many AWS outages has Netflix survived?

Re: Netflix is Down

#75

Earlier quoted context omitted.

If I remember correctly, the N. Virginia site is the default for EC2 instances, and is much much larger than the other availability zones. I've even heard that it's bigger than all the other zones combined. However, I'd love for someone to corroborate my statement.

That's certainly true for the public IP spaces. It's possible that the other regions have a higher proportion of instances in VPC which don't have public IP addresses, or a higher proportion of larger instance types, though.

Not all instance types are available in all zones, and some of the smaller ones are only available in this one.

Re: Netflix is Down

#76

Having been in a similar position in the past, I actually feel bad for all the people at amazon and netflix that'll need to work late tonight...

They are getting OT/Holiday pay I'm sure. So they will be ok.

Compensation isn't actually the point here...

Re: Netflix is Down

#77
post #58

Earlier quoted context omitted.

> When you build your own system, you decide how reliable you want it. Ha! You mean that you discover how hard and expensive it is to build your own reliable infrastructure or how hard it is to hire people to do it for you.

No offense, but I used to work at a DoE lab on data-taking from the Large Hadron Collider CMS detector. It's not as expensive as you'd think. We were dealing with hundreds of petabytes of data, and customers much more demanding than your consumer paying $16/month to stream movies.

How big was your team? How much were your IT costs?

Re: Netflix is Down

#78
post #69

Netflix is down because Amazon is down, so we had to watch shows on Amazon (prime) instead. Funny how that works!

I had the same experience. When watching movies with my SO tonight, I explained that Netflix uses AWS and that AWS was experiencing problems. We started watching a movie on Amazon prime and she asked "So I guess Amazon Video isn't using AWS?"

I tried to watch a movie rented from Amazon over Roku last night (not Prime) and it was unwatchable. It would get stuck on some scene for seconds to minutes, occasionally letting loose a bit of audio.

The movie was Inception. Maybe there's some irony in there.

Re: Netflix is Down

#79
post #53
post #42

Earlier quoted context omitted.

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

You rarely hear of Google outages http://techcrunch.com/2012/12/10/gmail-experiences-a-widespr... ,or iTunes http://appleinsider.com/articles/12/11/19/itunes-match-down-... ,or Youtube http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-... Everyone goes down, but EC2 has been buggy as hell from what I can tell. And before everyone says Netflix should `just` move or add more bandwidth... http://www.nbcnews.co…

That bandwidth for delivering video is being streamed from a CDN outside AWS.

Re: Netflix is Down

#80
post #77

Earlier quoted context omitted.

No offense, but I used to work at a DoE lab on data-taking from the Large Hadron Collider CMS detector. It's not as expensive as you'd think. We were dealing with hundreds of petabytes of data, and customers much more demanding than your consumer paying $16/month to stream movies.

How big was your team? How much were your IT costs?

10 people; 4 people on the actual sysadmin/operations side, 6 people who managed the higher application layer for the distributed filesystem, job management, etc.

If you're just talking the spinning disk and servers, probably ~$5-6MM/year. If you include the StorageTek tape archives, add in another $5MM-10MM.

Netflix operating income for FY2011 was $376 million.

Post reply on HN