Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix is Down

gigaom.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Netflix is Down

#51

Post by cpeterso »

campnic wrote: post #24
smoyer wrote: post #20

Soon it will be news when the N. Virginia services are actually up! I don't want to be too hard on Amazon because what they've built is pretty amazing, but I really have to wonder what's different about the N. Virginia site ... And why they're not having similar failures at the other sites.

Based on some rough second hand estimation [1] it is an order of magnitude larger then all other AZs and is about as big as them all combined. [1]: https://huanliu.wordpress.com/2012/03/13/amazon-data-center-...

What is an "AZ"? Availability Zone? "AZ" is not a very Google-friendly acronym. <:)

Re: Netflix is Down

#53

Post by pixl97 »

moe wrote: post #42
jrockway wrote: post #41

Earlier quoted context omitted.

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering…

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

You rarely hear of Google outages

http://techcrunch.com/2012/12/10/gmail-experiences-a-widespr...

,or iTunes

http://appleinsider.com/articles/12/11/19/itunes-match-down-...

,or Youtube

http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-...

Everyone goes down, but EC2 has been buggy as hell from what I can tell.

And before everyone says Netflix should `just` move or add more bandwidth...

http://www.nbcnews.com/technology/technolog/netflix-uses-32-...

They represent a huuuge amount of bandwidth usage!

Re: Netflix is Down

#54

Post by res0nat0r »

moe wrote: post #42
jrockway wrote: post #41

Earlier quoted context omitted.

The gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering…

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

>Especially one as trivial as Netflix which is largely read-only.

I stopped reading after this, since you obviously dont understand how complex their architecture really is.

Re: Netflix is Down

#55

Post by sbov »

res0nat0r wrote: post #54
moe wrote: post #42

Earlier quoted context omitted.

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

>Especially one as trivial as Netflix which is largely read-only. I stopped reading after this, since you obviously dont understand how complex their architecture really is.

I'm never one to trivialize another company's problems so would hesitate to say Netflix's case is easy, but you cannot counter this statement by saying their architecture is complex. I can build a complex architecture for "Hello World" if I wanted to. Complex architecture != complex problem.

Re: Netflix is Down

#56

Post by toomuchtodo »

pixl97 wrote: post #53
moe wrote: post #42

Earlier quoted context omitted.

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

You rarely hear of Google outages http://techcrunch.com/2012/12/10/gmail-experiences-a-widespr... ,or iTunes http://appleinsider.com/articles/12/11/19/itunes-match-down-... ,or Youtube http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-... Everyone goes down, but EC2 has been buggy as hell from what I can tell. And before everyone says Netflix should `just` move or add more bandwidth... http://www.nbcnews.co…

So work with eyeball networks and move your gear to the edge; dumping it all in AWS and hoping for the best is failing (at best). Do all the auth/billing/recommendations/transcoding in AWS, and push all the video content out to boxes at the ISPs.

Re: Netflix is Down

#57

Post by TkTech »

cpeterso wrote: post #51
campnic wrote: post #24

Earlier quoted context omitted.

Based on some rough second hand estimation [1] it is an order of magnitude larger then all other AZs and is about as big as them all combined. [1]: https://huanliu.wordpress.com/2012/03/13/amazon-data-center-...

What is an "AZ"? Availability Zone? "AZ" is not a very Google-friendly acronym. <:)

You guessed correct!

Re: Netflix is Down

#58

Post by herge »

toomuchtodo wrote: post #49
superuser2 wrote: post #44

Earlier quoted context omitted.

Because systems only crash when they're maintained by other people?

Assumption: Your business uses Amazon's EBS (because heh, you need to store your data somewhere between EC2 instance reboots/creation/destruction). EBS is down. What do you do? Hope AWS engineers get it back up fast? You can't do a damn thing about its reliability; you're stuck with whatever reliability Amazon has decided to deliver (which I think we can agree is much less than a company with revenue can depend on).…

> When you build your own system, you decide how reliable you want it.

Ha! You mean that you discover how hard and expensive it is to build your own reliable infrastructure or how hard it is to hire people to do it for you.

Re: Netflix is Down

#59

Post by moe »

res0nat0r wrote: post #54
moe wrote: post #42

Earlier quoted context omitted.

You're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix…

>Especially one as trivial as Netflix which is largely read-only. I stopped reading after this, since you obviously dont understand how complex their architecture really is.

I work for a media streaming site (rank "Trivial" may sound harsh but in terms of large scale systems it really doesn't get much simpler than serving static files. Obviously their auxiliary services (billing, content ingestion, social etc.) are nowhere near trivial. But there's no reason for the core services that are required to get the catalog and video to most[1] devices to not be damn near 100% available.

I'm not talking down netflix for having an outage anyway. Shit happens and afaik their overall track record is not bad at all. I only replied to jrockway's claim that multi-datacenter redundancy would require inconceivable engineering effort or amounts of money - neither is true.

[1] Excluding those that need realtime transcoding, absurd DRM schemes or similar.

Re: Netflix is Down

#60

Post by malyk »

zrail wrote: post #17
alxndr wrote: post #16

Earlier quoted context omitted.

heroku.com is up for me (near NYC) as are the Heroku-hosted sites we have at work.

It's not every app for me either. It appears to be a subset of apps that use the ssl:endpoint add-on to implement HTTPS.

And its not all sslenpoint sites either. We've been up through the whole thing. Which is critical seeing as we are a gifting company that provides an easy last minute gift!
Post reply on HN