Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Open Connect Appliances

gizmodo.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Netflix's Open Connect Appliances

#21

Post by pfg »

nrzuk wrote: post #14

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the r…

I'm no expert either, but if my napkin math is right, fully utilized in a best-case scenario (no network overhead, etc.), those disks would peak at around 30-40 MByte/s (36 disks, 10 Gbit NIC).

I'm not sure if the I/O is really all that random. Theoretically, it's all sequential, but might effectively hit the disk like random I/O because of concurrent streaming sessions, varying network speed and so on. My guess is you can get pretty close to sequential I/O with some simple means like using large block sizes in your RAID, well-tuned readahead and Native Command Queuing.

Re: Netflix's Open Connect Appliances

#22

Post by pfg »

bashinator wrote: post #18

I'd be interested to find out more of hardware details of these devices. 100+ TB of storage in a 4U is pretty respectable. From the images in the article, it looks like the drives are not hot-swappable, so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. For comparison, Supermicro makes one of the highest-density storage servers that I'm awa…

There's some info on the hardware on Netflix Open Connect site [1]. They mention it's heavily influenced by Backblazes Storage Pod[2].

[1]: https://www.netflix.com/openconnect/hardware [2]: http://blog.backblaze.com/2014/03/19/backblaze-storage-pod-4

Re: Netflix's Open Connect Appliances

#23

Post by josho »

wmf wrote: post #11
ubercore wrote: post #2

Interesting to see some more details on the appliances. Isn't the argument primarily about peering, not open connect appliances, though?

It's all one discussion in my mind. There are three ways to get Netflix: boxes, peering, or transit (from cheapest to most expensive). If peering saves the ISP money and the boxes save even more money, why are some ISPs refusing to use them?

Because some ISPs are trying to maximize their revenue by charging Netflix directly.

There are a few other legitimate reasons as well. Some ISPs dislike hosting hardware that they don't own in their data centre. Or it may be that an ISP has contracts in place that commit the ISP to certain bandwidth costs, so removing Netflix traffic may not actually save them money.

Re: Netflix's Open Connect Appliances

#24

Post by fragmede »

nrzuk wrote: post #14

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the r…

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible.

* Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives,

* Use 50 in a chassis of them instead of 12.

* Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid.

* Have multiple copies of a show on disk, as the article states happens.

* Optimize for a read-heavy workflow during peak hours, (eg, mount noatime).

The devil is in the details, but what I listed above is probably a good starting point. The article states a data rate of 3GB/hour, which is only .83 megabytes/second, * 100 streams is only 83 megabytes/second, which is easy for the above configuration. Hell, I'll bet the above configuration could do 10,000 streams if the data rate is 3GB/hour with no issues given Netflix's peak read-only workload.

Re: Netflix's Open Connect Appliances

#25

Post by jedberg »

bashinator wrote: post #18

I'd be interested to find out more of hardware details of these devices. 100+ TB of storage in a 4U is pretty respectable. From the images in the article, it looks like the drives are not hot-swappable, so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. For comparison, Supermicro makes one of the highest-density storage servers that I'm awa…

> It looks like the drives are not hot-swappable,

That's correct.

> so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed.

Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.

Re: Netflix's Open Connect Appliances

#26

Post by frio »

Isn't this just a CDN? ISPs quite happily host Akamai and Google nodes -- are they being paid for maintenance of those? Empirically, from an old job, the answer is no (in fact, we had to petition for quite a while to get some Google boxen), but that's in a country at the tail end of a long, long pipe. Is it different in the US?

Re: Netflix's Open Connect Appliances

#27

Post by osi »

JTon wrote: post #8

Can anyone explain why an ISP would push back Netflix installing one of these boxes? Does the ISP want Netflix service congestion to better sell its own media assets/distribution (i.e. cable TV)?

From the operator of a smaller ISP:

http://www.listbox.com/member/archive/247/2014/07/search/bmV...

Re: Netflix's Open Connect Appliances

#28

Post by tzs »

If the ISPs are reclassified as common carriers, will they still be allowed to let Netflix install those boxes?

Presumably there is limited space in the ISP data center, so they cannot let everyone who wants to stick a box in there do so, so I'm wondering what methods a common carrier can use to decide which content providers can have access, and how this fundamentally differs from the "fast lanes" that common carrier status is supposed to prevent.

Re: Netflix's Open Connect Appliances

#29

Post by acdha »

osi wrote: post #27
JTon wrote: post #8

Can anyone explain why an ISP would push back Netflix installing one of these boxes? Does the ISP want Netflix service congestion to better sell its own media assets/distribution (i.e. cable TV)?

From the operator of a smaller ISP: http://www.listbox.com/member/archive/247/2014/07/search/bmV...

That's Brett Glass, though, who is … opinionated. It's a good idea to read the entire thread for responses from a number of other network operators as well as some Netflix technical staff:

http://seclists.org/nanog/2014/Jul/index.html#265

Re: Netflix's Open Connect Appliances

#30

Post by wmf »

tzs wrote: post #28

If the ISPs are reclassified as common carriers, will they still be allowed to let Netflix install those boxes? Presumably there is limited space in the ISP data center, so they cannot let everyone who wants to stick a box in there do so, so I'm wondering what methods a common carrier can use to decide which content providers can have access, and how this fundamentally differs from the "fast lanes" that common carrie…

Maybe they can come up with a rule like the box must save 2 Gbps of peak hour bandwidth per U.

From the ISP's perspective they are hosting these boxes to save money, not improve performance. If performance happens to improve (ahem) that's a convenient side effect.

Post reply on HN