Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Open Connect Appliances

gizmodo.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Netflix's Open Connect Appliances

#31

Post by bashinator »

jedberg wrote: post #25
bashinator wrote: post #18

I'd be interested to find out more of hardware details of these devices. 100+ TB of storage in a 4U is pretty respectable. From the images in the article, it looks like the drives are not hot-swappable, so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. For comparison, Supermicro makes one of the highest-density storage servers that I'm awa…

> It looks like the drives are not hot-swappable, That's correct. > so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.

> Exactly. If a drive dies, the capacity of that box is just reduced.

So not even redundancy? Just cope with losing whatever media happened to be on the dead drive?

Re: Netflix's Open Connect Appliances

#32

Post by wmf »

frio wrote: post #26

Isn't this just a CDN? ISPs quite happily host Akamai and Google nodes -- are they being paid for maintenance of those? Empirically, from an old job, the answer is no (in fact, we had to petition for quite a while to get some Google boxen), but that's in a country at the tail end of a long, long pipe. Is it different in the US?

People don't know how CDNs work so it looks amazing.

Comcast and Verizon now claim that Akamai is indeed paying them. Perhaps the outcry over Netflix being extorted explains why these deals have remained secret for years.

Re: Netflix's Open Connect Appliances

#33

Post by thrownaway2424 »

fragmede wrote: post #24
nrzuk wrote: post #14

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the r…

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible. * Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives, * Use 50 in a chassis of them instead of 12. * Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid. * Have multiple copies of a show on disk, as the article states happens. * Optimize for a read-…

A 15000RPM SAS drive would be an _extremely_ poor choice for this type of workload. Power hungry, hot, extremely sensitive to vibration, very low data density, and, of course, ridiculously expensive.

Re: Netflix's Open Connect Appliances

#34

Post by wmf »

bashinator wrote: post #31
jedberg wrote: post #25

Earlier quoted context omitted.

> It looks like the drives are not hot-swappable, That's correct. > so I'd guess Netflix is able to remotely track loss of redundancy and will just send out an entire replacement unit when needed. Exactly. If a drive dies, the capacity of that box is just reduced. Once enough drives die, the box gets an RMA.

> Exactly. If a drive dies, the capacity of that box is just reduced. So not even redundancy? Just cope with losing whatever media happened to be on the dead drive?

They can always download another copy onto a working drive.

Re: Netflix's Open Connect Appliances

#35

Post by fragmede »

thrownaway2424 wrote: post #33
fragmede wrote: post #24

Earlier quoted context omitted.

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible. * Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives, * Use 50 in a chassis of them instead of 12. * Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid. * Have multiple copies of a show on disk, as the article states happens. * Optimize for a read-…

A 15000RPM SAS drive would be an _extremely_ poor choice for this type of workload. Power hungry, hot, extremely sensitive to vibration, very low data density, and, of course, ridiculously expensive.

What? While I can't argue with them being ridiculously expensive (because they very much are) why would them being power hungry, hot, and extremely sensitive to vibration even matter if they are going to be rack-mounted into an ISP's data center, with redundant power-supplies and a data-center grade cooling system to match?

The only possible issue of the ones you raised is low data density, but all engineering is a trade-off, 15k rpm drives get you better seek times, which would generally lead to the ability to support more viewers in a single box - not working for Netflix, I don't know how many users they want to support per-box.

It's totally possible that between the number of streams they want to support, and the total storage in the box that they've gone with SATA drives, but to pretend that 15k RPM SAS drives are "an _extremely_ poor choice" for an enterprise-grade storage system would be to ignore the fact that Netflix is making an enterprise-grade storage appliance - and that on the top-end, those appliances commonly use SAS drives.

Re: Netflix's Open Connect Appliances

#36

Post by ssmoot »

pessimizer wrote: post #6
Thaxll wrote: post #5

Lot of mistakes in this article, afair it can't hold all the movies but just the most popular. http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pd... " Each appliance stores a portion of the catalog, which in general is less than the complete content library for a given region. Popularity changes, new titles being added to the service, and re- encoded movies are all part of the up to 7.5TB of nightly updates…

"But wait, you say. That'd be all fine and dandy if Netflix was etched in stone, but it's not! They add and remove stuff all the time! That's right, which is why most OCAs take a 7.5TB update. Every. Freaking. Day. "If you had to take a 7.5TB update on your home internet, you would be screwed. According to Ookla's Net Index, the average download speed in the United States is 18.6Mbps. So with an average connection, t…

I'm sure thought has been put into it. I just wonder why the OCAs aren't caches? I'd imagine a cache you be a lot more (mostly bandwidth) efficient for the ISPs and scale much better.

I request a stream. It's proxied through the device. It's not present on the device. So the device requests the 1080P from Netflix. It transcodes the stream it receives on-the-fly and sends me whatever quality I require while writing both the original and my lower-quality version to disk.

I mean maybe the limitation in that plan is the number of streams that could be transcoded simultaneously. So maybe instead the on-the-fly version isn't cached. Because I may need to switch quality on-the-fly. So maybe instead jobs are just queued up for the 5 or so formats required based on the 1080P download and they can process on spare transcoders opportunistically.

Just interesting to think through it as a programming problem. I'm sure the people working on this are very smart. It just seems really heavy on the bandwidth utilization as is.

Re: Netflix's Open Connect Appliances

#37

Post by cjg_ »

fragmede wrote: post #24
nrzuk wrote: post #14

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the r…

There's a bit of magic to it, turning the OS to work well in conjunction with the hardware you have, but it's possible. * Use 15k RPM SAS 12Gb/s drives instead of cheap consumer drives, * Use 50 in a chassis of them instead of 12. * Use RAID 0 instead of RAID 5 or 6 - I'll be the quoted storage space is raw, not post-raid. * Have multiple copies of a show on disk, as the article states happens. * Optimize for a read-…

They are actually using standard SATA drives, see spec here, https://www.netflix.com/openconnect/hardware

Re: Netflix's Open Connect Appliances

#38

Post by thrownaway2424 »

fragmede wrote: post #35
thrownaway2424 wrote: post #33

Earlier quoted context omitted.

A 15000RPM SAS drive would be an _extremely_ poor choice for this type of workload. Power hungry, hot, extremely sensitive to vibration, very low data density, and, of course, ridiculously expensive.

What? While I can't argue with them being ridiculously expensive (because they very much are) why would them being power hungry, hot, and extremely sensitive to vibration even matter if they are going to be rack-mounted into an ISP's data center, with redundant power-supplies and a data-center grade cooling system to match? The only possible issue of the ones you raised is low data density, but all engineering is a t…

I will double-down to claim that 15krpm SAS drives are a bad choice for any application, and they are only used as a bandaid for marginal improvements on irredeemable system designs.

To address your points individually:

1) Power hungry. When you add in conversion and distribution and cooling, every watt consumed by the computer is consumed again by the datacenter infrastructure. Power costs money.

2) Hot is just the corollary of 1). Hot is also the enemy of density, and this box is very dense.

3) Sensitive to vibration. If you aren't intimately familiar with this fact then you aren't getting the performance you paid for from your 15k disks. To achieve their spectacular claimed seek times then need very careful mechanical design of their enclosure. Much more careful than racking up one of Netflix's boxes in a rack with other random vibrators.

4) Density. To get the space Netflix is using here, you need 5x more expensive 15k disks because they top out at 600GB and the ones people actually use for these workloads have 3TB.

A smart read-ahead strategy obviates the need for shiny seek time specs. For any given stream you could read ahead by 32MB or whatever. Now you've made seek time irrelevant. Put lots of RAM in the machine and you're done at a tiny fraction of the capital and operating costs of 15k disks.

Re: Netflix's Open Connect Appliances

#39

Post by justincormack »

pfg wrote: post #21
nrzuk wrote: post #14

Could someone explain to me (it's been a long day, and I'm no storage expert!) how these are useful, wouldn't you need a whole bunch of these at each location they are to be installed? Just basing this on my own experience I have 12 drives in RAID which have a fairly substantial sequential throughput. Start multiple streams of high bandwidth videos and the maximum throughput from the drives drops sharply due to the r…

I'm no expert either, but if my napkin math is right, fully utilized in a best-case scenario (no network overhead, etc.), those disks would peak at around 30-40 MByte/s (36 disks, 10 Gbit NIC). I'm not sure if the I/O is really all that random. Theoretically, it's all sequential, but might effectively hit the disk like random I/O because of concurrent streaming sessions, varying network speed and so on. My guess is y…

The new versions are all SSD.
Post reply on HN