Netflix is now doing per-shot encoding for UHD content • Hacker News Board

Re: Netflix is now doing per-shot encoding for UHD content

#51

Post by jiggawatts » Fri, Aug 28, 2020, 11:07 PM UTC

csdreamer7 wrote: post #17

arendtio wrote: post #12

Looks amazing and while I would like to enjoy such quality and am willing to pay for the premium plan (in fact I do already), I still can't watch the 4K content even in a world where I accept DRM modules in my browser, because some DRM plugins seem to be more equal than others.

Do we still need an Intel Kaby Lake or later CPU for 4K DRM decoding? (And Windows... forgot about that.) If so, Netflix is on the wrong side of the CPU war with AMD ascending.

The arguments Netflix made for this highly model-specific DRM never made sense to me. There are 4K rips of pretty much every Netflix show available for download on various bays related to piracy.

So why bother locking your paying customers out of 4K content? What's the upside?

Also: I'm reasonably convinced that NetFlix is violating the law, at least in Australia, by advertising that their content is 4K and then arbitrarily blocking access to the 4K streams. This is called "bait & switch", and the fines are eyewatering, even for large corporations.

Re: Netflix is now doing per-shot encoding for UHD content

#52

Post by Groxx » Fri, Aug 28, 2020, 11:07 PM UTC

myself248 wrote: post #41

Groxx wrote: post #16

Earlier quoted context omitted.

Coarsely detecting cuts is relatively simple - look for large frame-to-frame differences (e.g. encode however -> find large frames surrounded by smaller ones -> done, it's as accurate as your perceptual compression is). There are a number of ffmpeg-using tools out there doing this and other "cut to / from black" detection and it's pretty good. Not good enough for a human to say "yeah, these are all scenes", but proba…

Seems to me that if the frame-to-frame difference isn't big enough to detect that way, it's not likely to benefit from a new I-frame, yeah?

That's the basic idea, yeah. It falls apart in a couple places, e.g. when the cut or fast-fade goes to a very cheap frame like a mostly solid color, and it may not detect stuff like whip-cuts (since a whole chunk of frames are expensive), but so many scenes in so much of media has single-frame cuts that it's well within that "good enough" range.

And for dynamic encoding like this: when it's wrong, it's not visually worse in that scene than choosing that sub-par encoding for the entire movie, which has the same "choose the best encoding" problem as individual chunks have. I assume it'd be relatively rare for it to result in anything worse than a one-shot strategy.

---

ffmpeg will let you easily do frame-to-frame-diff logic that lets you chop videos into scenes, for example: https://video.stackexchange.com/a/30701 I'm not sure how much it handles compressed-frame differences, but it shouldn't be too hard to build around it. Just might be a bit beyond bash-friendly.

Re: Netflix is now doing per-shot encoding for UHD content

#53

Post by mappu » Fri, Aug 28, 2020, 11:08 PM UTC

What exactly is shot-based encoding doing here?

Even old XviD would reliably always insert a new I-frame / start a new GOP on a scenecut, and perform global rate optimization based on scene complexity within the target ABR parameters.

Re: Netflix is now doing per-shot encoding for UHD content

#54

Post by sbierwagen » Fri, Aug 28, 2020, 11:09 PM UTC

ProfessorLayton wrote: post #39

radley wrote: post #20

> Sometimes we ingest a title that would need more bits at the highest end of the quality spectrum — even higher than the 16 Mbps limit of the fixed-bitrate ladder. From just a cursory inspection using my Apple TV 4K, 12 Mbps is their target bitrate but their ceiling was ~16 Mbps. That's still only a third of what AppleTV+ offers (36/48) and only 2/3 of Disney+ (18/24). Netflix is still higher than Prime Video (10/14…

I've noticed that ATV+ looks the best, which is much appreciated. However, what has been driving me insane with all these services is how the bitrate is completely inconsistent throughout, depending on network congestion. Every service I subscribe to will "automagically" lower the bitrate if the network can't handle it. Which is fine, I get it. The thing is, it's the only way to watch anything, and depending on the c…

Are you streaming over wifi?

Re: Netflix is now doing per-shot encoding for UHD content

#55

Post by zucker42 » Fri, Aug 28, 2020, 11:10 PM UTC

jiggawatts wrote: post #51

csdreamer7 wrote: post #17

Earlier quoted context omitted.

Do we still need an Intel Kaby Lake or later CPU for 4K DRM decoding? (And Windows... forgot about that.) If so, Netflix is on the wrong side of the CPU war with AMD ascending.

The arguments Netflix made for this highly model-specific DRM never made sense to me. There are 4K rips of pretty much every Netflix show available for download on various bays related to piracy. So why bother locking your paying customers out of 4K content? What's the upside? Also: I'm reasonably convinced that NetFlix is violating the law, at least in Australia, by advertising that their content is 4K and then arbi…

I suspect the arguments are made to make sense to media executives not to the consumer. After all, DRM is an anti-feature for the consumer.

Re: Netflix is now doing per-shot encoding for UHD content

#56

Post by snailmailman » Fri, Aug 28, 2020, 11:11 PM UTC

ProfessorLayton wrote: post #39

radley wrote: post #20

> Sometimes we ingest a title that would need more bits at the highest end of the quality spectrum — even higher than the 16 Mbps limit of the fixed-bitrate ladder. From just a cursory inspection using my Apple TV 4K, 12 Mbps is their target bitrate but their ceiling was ~16 Mbps. That's still only a third of what AppleTV+ offers (36/48) and only 2/3 of Disney+ (18/24). Netflix is still higher than Prime Video (10/14…

I've noticed that ATV+ looks the best, which is much appreciated. However, what has been driving me insane with all these services is how the bitrate is completely inconsistent throughout, depending on network congestion. Every service I subscribe to will "automagically" lower the bitrate if the network can't handle it. Which is fine, I get it. The thing is, it's the only way to watch anything, and depending on the c…

On my phone it lets me download Apple TV+ content. The files are shockingly large compared to what Amazon Video downloads are. A 1 hr clip was maybe 800mb in prime, but a few gigabytes from apple. (I don’t have any on my phone atm, but a quick search is telling me episodes of “For all mankind” are ~4GB and an hour long) It looks noticeably better, a massive step up that is completely worth the extra bytes.

On my desktop I always rent stuff through iTunes just cause I can download it in advance, and don’t have to deal with buffering or reduced quality.

Re: Netflix is now doing per-shot encoding for UHD content

#57

Post by jdmichal » Fri, Aug 28, 2020, 11:11 PM UTC

wtallis wrote: post #34

hn_check wrote: post #25

Earlier quoted context omitted.

The chart does not see to show every bitrate for every target resolution, but rather is show where the 4K stream would have to drop to a lower resolution given insufficient bandwidth.

https://miro.medium.com/max/2000/1*c28F7YXjNo-GpmB9bBmbrA.pn... This chart clearly shows multiple data points per resolution under the old fixed-ladder scheme, and under the new scheme the new data point for 1080p is lower on the video quality scale than any of the old 720p encodes. That's a pretty significant difference, and my question about where these new encodes fit into their existing price structure remains va…

> As a side note, we do have some additional points, not shown in the plots, that are used in resolution limited scenarios — such as a streaming session limited to 720p or 1080p highest encoding resolution. Such points lie under (or to the right of) the convex hull main ladder curve but allow quality to ramp up in resolution limited scenarios.

They're showing the curve that is used for connections that want to be streaming 4k. So if you're on the 1080 stream in the charted scenario, it's because you're bandwidth-limited. They have higher-quality, higher-bitrate encoding curves for lower-resolution connections. However, it doesn't make sense to use that 1080 encoding for a 4k connection. If you could afford the bitrate, you'd rather be streaming the higher resolution from the displayed curve.

Re: Netflix is now doing per-shot encoding for UHD content

#58

Post by paxswill » Fri, Aug 28, 2020, 11:13 PM UTC

michaelt wrote: post #42

notJim wrote: post #31

Earlier quoted context omitted.

> From just a cursory inspection using my Apple TV 4K I'm confused, how are you getting this information? Does Apple TV have a way to display the bitrate ceiling? Also, do you find the bitrate differences noticeable? On my 4k tv, I find Netflix looks pretty good, but Amazon looks pretty bad and HBO Now is awful of course (no Max yet, since I use a Roku.) This might be enough to get me to check out AppleTV+.

> how are you getting this information? Does your router report its current throughput?

Not Apple TV specific, but Netflix (and I think Prime) have test videos that also show the bitrate as it’s streaming. I used them a while back when setting up my network and access point location.

Re: Netflix is now doing per-shot encoding for UHD content

#59

Post by dimnsionofsound » Fri, Aug 28, 2020, 11:13 PM UTC

sjwright wrote: post #36

charrondev wrote: post #5

That seems like a pretty impressive achievement! Question for anyone here: What do you use to play 4K/HDR? I have an Apple TV 4K, which can do 4K Dolby Vision playback and looks ok, but the Apple TV tends to have some jittering when streaming certain shows (very noticeable in panning shots of animation). The sound quality is also noticeably worse on my set with it (it doesn’t seem to be able to do direct pass through…

The jittering is caused by the Apple TV defaulting to 60p frame rate for all content. But if you turn on “match frame rate” the jittering goes away.

Before I turned that on, watching BBC documentaries in 25 FPS the judder was very noticeable on panning shots. Definitely worth enabling if your device supports it.

Re: Netflix is now doing per-shot encoding for UHD content

#60

Post by duhi88 » Fri, Aug 28, 2020, 11:14 PM UTC

purerandomness wrote: post #7

2bitencryption wrote: post #4

Earlier quoted context omitted.

Instead of one optimization profile for a whole movie, Netflix is detecting when the shot of a movie changes (the camera "cuts" from one shot to another) and beginning a new optimization profile specifically for the content of that shot. At least, that is my understanding.

That sounds so intuitive that I want to follow up with the question: Why isn't this the default? Is detecting a cut particularly difficult?

"Variable bitrate" encoding is a thing, which allows chunks of a file to be encoded at a higher or lower bitrate depending on how much is happening. I don't know exactly how it happens, but I assume that each chunk is determined by duration or size, while Netflix's new method determines each chunk by the content.

In order for this to be the default, you'd need either humans, or pattern recognition algorithms to identify the chunks. You also need to quantify how by how much each chunk needs its encoding parameters tweaked.

Monetary costs aside, that's increasing complexity of your pipeline with relatively small gains. I'd bet that more companies will start looking at similar approaches now that 4K HDR (and 8K) are becoming more common. Probably not worth the R&D for 1080p, but we'll see tricks like this start to trickle down, I'm sure.

It's highly unlikely we'll see anything similar in FOSS tools in the near future.