Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Open Connect Appliances

gizmodo.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Netflix's Open Connect Appliances

#41

Post by fragmede »

cjg_ wrote: post #37
fragmede wrote: post #24

Earlier quoted context omitted.

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible. * Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives, * Use 50 in a chassis of them instead of 12. * Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid. * Have multiple copies of a show on disk, as the article states happens. * Optimize for a read-…

They are actually using standard SATA drives, see spec here, https://www.netflix.com/openconnect/hardware

"The following system was developed and first deployed at the end of 2011."

Re: Netflix's Open Connect Appliances

#42

Post by MBCook »

bashinator wrote: post #31
jedberg wrote: post #25

Earlier quoted context omitted.

> It looks like the drives are not hot-swappable, That's correct. > so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.

> Exactly. If a drive dies, the capacity of that box is just reduced. So not even redundancy? Just cope with losing whatever media happened to be on the dead drive?

The box is effectively one giant cache. If you lose a drive you can get the movie back from the main distribution network.

The article also mentions the box stores multiple copies of done things for increased throughput on popular titles.

Re: Netflix's Open Connect Appliances

#43

Post by jsz0 »

JTon wrote: post #8

Can anyone explain why an ISP would push back Netflix installing one of these boxes? Does the ISP want Netflix service congestion to better sell its own media assets/distribution (i.e. cable TV)?

It may very well be in their best interest to give NetFlix free co-location space but because of the PR battle this has become kind of a proxy war for bigger issues. ISPs have a legitimate concern now that if NetFlix 'wins' the PR battle they will have a long line of companies knocking on their door demanding peering arrangements and free co-location. It's also worth noting ISPs could rent out 4RU of space in their facilities to other companies for way more than $0 and often do.

Re: Netflix's Open Connect Appliances

#44

Post by mturmon »

ssmoot wrote: post #36
pessimizer wrote: post #6

Earlier quoted context omitted.

"But wait, you say. That'd be all fine and dandy if Netflix was etched in stone, but it's not! They add and remove stuff all the time! That's right, which is why most OCAs take a 7.5TB update. Every. Freaking. Day. "If you had to take a 7.5TB update on your home internet, you would be screwed. According to Ookla's Net Index, the average download speed in the United States is 18.6Mbps. So with an average connection, t…

I'm sure thought has been put into it. I just wonder why the OCAs aren't caches? I'd imagine a cache you be a lot more (mostly bandwidth) efficient for the ISPs and scale much better. I request a stream. It's proxied through the device. It's not present on the device. So the device requests the 1080P from Netflix. It transcodes the stream it receives on-the-fly and sends me whatever quality I require while writing bo…

I can think of reasons why a pure cache would impose costs that are too high -- namely, the device would then be writing during times of very read load.

But: I agree with your bigger point, that it's an interesting programming problem.

We're all familiar with CPU caches, so it's interesting mental exercise to think of this as the same system, but optimized for a different part of design space.

Re: Netflix's Open Connect Appliances

#45

Post by porpoisemonkey »

frio wrote: post #26

Isn't this just a CDN? ISPs quite happily host Akamai and Google nodes -- are they being paid for maintenance of those? Empirically, from an old job, the answer is no (in fact, we had to petition for quite a while to get some Google boxen), but that's in a country at the tail end of a long, long pipe. Is it different in the US?

What this article is describing sounds like a colocation agreement which is a quite common service offered by ISPs and is typically paid for by a monthly rental fee calculated by storage size, dedicated bandwidth use and power consumption.

Re: Netflix's Open Connect Appliances

#46

Post by cjg_ »

fragmede wrote: post #41
cjg_ wrote: post #37

Earlier quoted context omitted.

They are actually using standard SATA drives, see spec here, https://www.netflix.com/openconnect/hardware

"The following system was developed and first deployed at the end of 2011."

Newer revisions use 4 or 6 TB drives and such 15K SAS drives does not exist from what I can find.

See http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd...

Re: Netflix's Open Connect Appliances

#47

Post by runlevel1 »

twoodfin wrote: post #19
nrzuk wrote: post #14

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the r…

These things can pump out 12Gbit during peak viewing, which is enough for about 4,000 avg. 3 Mbit streams. If they can cram in 256GB of DRAM, that's enough room for a 64MB buffer per stream, or about 170 seconds worth of streaming. Now you only need to be able to fill those buffers at a rate of about 24/second. I'm assuming whatever file system is on the disks uses a massive block size, so the number of seeks you'd h…

> I'm assuming whatever file system is on the disks...

Netflix uses UFS+J on FreeBSD 10.

Here are some notes from their talk at NYCBSDCon 2014:

- 400,000 stream files per appliance.

- 5,000 - 25,000 client streams per appliance.

- 300 - 500 streams coming off each disk all the time.

- Attempt to buffer 1MB ahead, but caching is futile.

- Result is completely random disk workload.

- System becomes limited by disk latency and CPU load.

Video here: https://www.youtube.com/watch?v=FL5U4wr86L4

Re: Netflix's Open Connect Appliances

#49

Post by qq66 »

josho wrote: post #23
wmf wrote: post #11

Earlier quoted context omitted.

It's all one discussion in my mind. There are three ways to get Netflix: boxes, peering, or transit (from cheapest to most expensive). If peering saves the ISP money and the boxes save even more money, why are some ISPs refusing to use them?

Because some ISPs are trying to maximize their revenue by charging Netflix directly. There are a few other legitimate reasons as well. Some ISPs dislike hosting hardware that they don't own in their data centre. Or it may be that an ISP has contracts in place that commit the ISP to certain bandwidth costs, so removing Netflix traffic may not actually save them money.

Why shouldn't an ISP charge Netflix to host a box in their datacenter? I'd charge you a fee if you wanted to store your DVDs in my house, even if you let me watch them.
Post reply on HN