Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix’s Media Production Suite

netflixtechblog.com

Post reply on HN
Print view View original

81–90 of 154 posts

Re: Netflix’s Media Production Suite

#81

Post by dkh »

fezz wrote: post #48
progbits wrote: post #32

Earlier quoted context omitted.

> we were backing up, two or three times So they just rediscovered what IT world knew for decades, or what am I missing?

When Data/File based workflows started in movies (around 2004), 2-3 copies was the standard from the get go and ideally this was with MD5 checksums (currently xxhash is more common because it's alot faster). LTO backups are also generally part of the copy chains as the 3rd or 4th copy. Before that, duplication with tape was while recording wasn't as common, but it was more common to duplicate after recording. Althoug…

3 different copies driven to 3 different places by 3 different people before you leave set for the day continued to be how it was when I was working on set. And believe it or not, there was still one incident in 2015 where Murphy's Law negated all 3 and I spent about a week file-carving the $60k worth of footage we didn't have the ability to reshoot again if we had to

Re: Netflix’s Media Production Suite

#82

Post by sagacity »

dkh wrote: post #77
_m_p wrote: post #29

I was reading about the cinematography of _Collateral_, possibly the first large budget feature film to be shot digitally, and one of the issues back in 2004 when it was made was the amount of storage required for digital video and the risk of not being able to retrieve the images from the data stores: > “We did massive testing with the hard drives, and everything was great, and then we had an experience where we sho…

Collateral was not the first fully digitally shot feature film. In fact, Collateral was not even fully digital. (The first major , all-digital, HD feature film was Attack of the Clones , but there were other fully-digital feature films before that, just not as major, and/or not always HD. Robert Rodriguez' Once Upon a Time in Mexico (2001) was fully digital.) But you are right that Collateral did do something very ne…

Around 2004 I worked for a company in The Netherlands that owned a Viper camera (one of the few in NL, I guess because they were based in Breda and Thomson had an HQ there). The company actually had a big Mercedes van that contained a Quantel iQ system just to record and postprocess the video coming out of that Viper.

In the years after that I worked with them to write a custom application based on a Bluefish444 card combined with some ATTO fiber channel storage just to get the frames to disk fast enough. A lot of custom code, overlapped I/O, that kind of thing. We had a beast of a JBOD RAID setup, must have been about 12 spinning disks.

The only alternative in those days were systems that stored to tape, but could only do so in a compressed format (I think Sony had a solution that did 4:2:0 instead of the 4:4:4 coming out of the Viper). People were scrambling for these storage solutions so much that we even got Arri to lend us their prototype D-20 camera (which turned into D-21 which turned into Alexa) just so we could make sure our storage system worked with their camera. We just had this amazing prototype camera sitting around our office for what must have been a year. They just lent it to us. Wild. I think our only main competitor at the time was Codex, which admittedly had a much slicker system.

We visited the CINEC trade show and got a ton of interest. I think I still have a business card of the DoP that did all the miniature work in Lord Of The Rings. He loved the fact that we would store things uncompressed, which would make things like compositing a lot easier.

Unfortunately, mismanagement caused the whole thing to collapse. Oh well. Nowadays you just use a CompactFlash card :)

Re: Netflix’s Media Production Suite

#83

Post by dkh »

pjc50 wrote: post #78
dkh wrote: post #77

Earlier quoted context omitted.

Collateral was not the first fully digitally shot feature film. In fact, Collateral was not even fully digital. (The first major , all-digital, HD feature film was Attack of the Clones , but there were other fully-digital feature films before that, just not as major, and/or not always HD. Robert Rodriguez' Once Upon a Time in Mexico (2001) was fully digital.) But you are right that Collateral did do something very ne…

I saw Russian Ark! Definitely a piece of art made by film buffs for film buffs; impressive to see, but far more impressive when you understand the amount of work that went into it. I'm wondering why people would have chosen to do early digital if it was so inconvenient. When did the cost and flexibility advantages start to really kick in?

So in the context of this specific goal—shooting a feature film in a single unbroken shot—digital was a pain in the ass, but this was close to impossible to do on film, and Russian Ark was the first to ever do it, on any medium.

Simply shooting a feature film digitally was not that complicated by this time, or at least it didn't have to be. The Sony CineAlta F900 was the camera developed to shoot the Star Wars prequels, and was revolutionary at the time, became the gold standard for years, and very convenient relative to film. Tons of things started to be shot in 1080p around that time, and it was very nice to work with. Collateral was insane because they wanted to shoot raw and at high frame rates. Russian Ark needed a single unbroken shot in a form factor that one human would be capable of holding for that long. Aside from very specific and/or boundary-pushing needs, the arrival of the F900 in 2000 was effectively when digital was more convenient than shooting film while also meeting the technical requirements of high-end production (though it was many more years before most cinematographers agreed that the image quality was comparable)

Re: Netflix’s Media Production Suite

#84

Post by dkh »

MarceliusK wrote: post #61
BhavdeepSethi wrote: post #53

15 years ago, the first start up I worked for provided APIs for music streaming in India. One of the founders who managed all infra was in US, and so the servers (bare metal) were in LA. I still find it amusing, that it was cheaper (and faster) just to fly to India, buy bunch of portable hard drives, upload the media, fly back to US and upload the data to the file server, than uploading the media directly from India…

How much global infrastructure has improved… but also how physical logistics can still beat the internet when you're dealing with massive datasets.

The technical requirements always seem to increase at the same rate as the technical advances. I've found this especially true in film/TV. Sure, by 2014ish we were shooting on solid state and had giant RAIDs on set and storage was cheaper than it ever had been, but we easily negated all of that by shooting on multiple RED cameras in raw at resolutions of 6.5k+. Terabytes of new data each day, even duplicating it before leaving took a lot of time! And then storing it at the office while letting more than 1 editor work with it at the time meant building a 36-disk ZFS server with 10GbE to each client. Just playing the footage back on a computer required a dedicated PCIe card

Re: Netflix’s Media Production Suite

#85

Post by ahmedfromtunis »

mrandish wrote: post #45
thecybernerd wrote: post #36

I wonder if Netflix will offer MPS as a service in the future for shows that are not on Netflix.

Probably not. One reason is that this is a competitive advantage for Netflix but I suspect the larger reason is that they created this to optimize their primary business, while operating it for other productions would be a secondary business and not mission critical. Another reason is that productions not on Netflix have other commercial options for camera to cloud production workflows (Frame.io for example). This wo…

> I suspect the larger reason is that they created this to optimize their primary business

Wasn't this how AWS started at Amazon as well?

Re: Netflix’s Media Production Suite

#86

Post by ahmedfromtunis »

prmoustache wrote: post #74
echelon wrote: post #4

Having a dozen different VFX departments using different file transfer methods like FTP seems like a nightmare. But then I realized that the banks do this, and probably worse. There's one that uses Gmail to exchange documents (not financial, but important nonetheless) and uses the read receipt to determine if it has ingested the data. Replaying ingestion is marking unread.

I've seen people in healthcare send patient data through whatsapp to other doctors in the same hospital. It was not allowed and they had reliable corporate tools to exchange patient data, but the UX probably felt so cumbersome they'd rather face legal risks than doing it the right way.

It doesn't make it any less worse, but that was my takeaway from the Signal scandal. Of course more secure channels exist, but I'm sure the UX sucked big time!

Re: Netflix’s Media Production Suite

#87

Post by dkh »

MarceliusK wrote: post #58

The part that struck me most was how much manual, error-prone work is still common in the industry. Still, I wonder how portable this is outside Netflix. It sounds like a very vertically integrated solution.

There is still a lot of manual, error-prone work, even at Netflix. Netflix just has workflows that ensure so many passes are of each task are done, with so many failsafes, fallbacks, tests, checks (both human and automated), and they start doing them all earlier in the production process than most other studios so they can handle things they still didn't account for.

Yes, it's pretty vertical, but essentially every big streaming platform or production company is. There's a very specific Netflix way that governs how all Netflix shows are produced. There is a very different but still meticulously standardized way that governs how all Hulu shows are produced, one for all Warner Bros. shows, etc. This is an area where being vertically integrated is totally fine. It not only makes enormous sense for these studios, but nobody outside of those environments wants or needs these workflows. Netflix's workflows are there to aid Netflix even more than their shows, and while there's a lot of excellent stuff in their workflows that most productions should utilize for efficiency/safety/whatever, there's also a ton of stuff that would make no sense to use independently.

Re: Netflix’s Media Production Suite

#88

Post by panick21_ »

fastball wrote: post #43
anigbrowl wrote: post #40

Earlier quoted context omitted.

I mean you could say the same thing about video cameras making it too easy to shoot footage compared to film.

Yes, and it is arguably true, right? That doesn't mean digital shouldn't exist, but everything is a balance. The quality and effort put into the "average" movie has almost certainly gone down over the decades since the normalization of digital cameras. If our ability (as a society or as individuals) to filter out the slop from the rest increases in lock-step, this is a non-issue. But it seems that this not what has h…

That's just your avg elitism. The idea that the world is better by making the creation of art expensive so that only a small group of elite people is nonsense.

Effort of course went down that's clearly a good thing. As for quality, before you needed to get enough return to pay for the expensive equipment and process, so likely you only do it for very few project. So on some abstract sense maybe 'quality' did go down, but that isn't bad as the total amount of high quality goes up far more.

See the massive amount of great TV that have happened since digital cameras.

> slop

What's slop for you is somebody else's favorite show.

Art doesn't have to 'impact us in a positive way' whatever that means. You are not a better person for having watched "Lawrence of Arabia". And in the past most people didn't watch "Lawrence of Arabia" but generic TV shows.

Personally I can easily filter the 'slop' (ie thinks I don't care about) so given how much better the ability is to select what to watch. On demand media libraries, recommendation systems, digital word of mouth and so on. In the past there were few TV channels and only a few movies in theaters at the same time.

So the total access to high quality content has gone up exponentially.

Re: Netflix’s Media Production Suite

#89

Post by barrkel »

gcanyon wrote: post #33

I read through that whole article thinking, - I wonder what the UI looks like compared to tools I use now - I wonder if there will be a free tier, since my video needs are modest It never occurred to me until I reached the end that this wasn't a "enjoy this tool we made" post, but instead a "look how awesome we are" post. :-/

The purpose of these articles is to promote the brand among engineers, help hire engineers, and help the careers of the authors both internally ("I'm helping the company hire") and externally (you can point at what you built because it's now public).
Post reply on HN