Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time, in-camera background compositing in The Mandalorian

ascmag.com

Post reply on HN
Print view View original

61–70 of 126 posts

Re: Real-time, in-camera background compositing in The Mandalorian

#61

Post by kragen »

mitchelhall wrote: post #50

Hi all, really cool that you have taken an interest in this project, a lot of your comments below are very insightful and interesting. I work on the team that deploys this tech on set. We focus on how the video signals get from the video output servers to the LED surfaces, coordinating how we deal with tracking methods, photogrammetry, signal delivery, operator controls and the infrastructure that supports all of thi…

Thanks! This work is really inspiring! Are these OLED screens, matrices of separate LEDs of the usual InGaAs and GaN type, or LCDs backlit with LEDs? The 2.84 mm pixel pitch makes it sound like it's separate inorganic LEDs.

Are there times short of sunshine where you need more directionality to the lighting than the screens can provide? Because the screens can emit from any direction but not toward any direction, being quasi-Lambertian emitters.

Re: Real-time, in-camera background compositing in The Mandalorian

#62

Post by m3at »

mitchelhall wrote: post #50

Hi all, really cool that you have taken an interest in this project, a lot of your comments below are very insightful and interesting. I work on the team that deploys this tech on set. We focus on how the video signals get from the video output servers to the LED surfaces, coordinating how we deal with tracking methods, photogrammetry, signal delivery, operator controls and the infrastructure that supports all of thi…

Great work! It must be a tremendously interesting job.

You might be able to answer my question which is: why use exclusively LED and no projector? I imagine that it's mainly because it's too dim and the main goal is to get good reflections. Is that something that was considered?

(I'm wondering as I found the work of teamLab very impressive, which rely heavily on projectors: https://www.teamlab.art/)

Re: Real-time, in-camera background compositing in The Mandalorian

#63

Post by BubRoss »

vernie wrote: post #34
RaptorJ wrote: post #15

Earlier quoted context omitted.

From the video posted upthread, around 3:40 it looks like they're doing just that: https://youtu.be/gUnxzVOs3rk?t=220

The most interesting aspect to me is that the system is pulling double duty by displaying both a dynamic, perspective-correct backdrop for the camera's POV and a static view for environment lighting and reflections outside of the camera's view frustum. I wonder if they had to take care to mitigate artifacts caused by the dynamic view bouncing off of surfaces facing away from the camera.

What artifacts?

Re: Real-time, in-camera background compositing in The Mandalorian

#64

Post by flashman »

kragen wrote: post #39
flashman wrote: post #29

I wonder how the photogrammetry aspect will intersect with intellectual property laws. The example used - scanning in a Santa Monica bar so that you can do reshoots without revisiting the location - would be an obvious example that might raise someone's hackles ("because it's our bar you're using to make your money" for instance). If you add that bar to your digital library, do you have to pay them royalties each tim…

If you have to pay royalties, they won't be to the bar; they'll be to the bar's architect. Copyright law generally only covers expressive, rather than functional, aspects of a copyrighted work, so things like doors and walls might be okay, but architectural design is recognized as copyrightable. In general I strongly recommend avoiding the term "intellectual property" because it conflates several different areas of l…

That's why I chose the term "intellectual property": it's a useful term for the issues around "creating a thing that's somewhat like another thing," even if it's not a super helpful legal term.

I would not be surprised to see someone argue for a new class of property rights for owners, surrounding the reproduction of a scene or location where the reproduction has commercial value. What happens if Facebook bootlegs the Mall of America for Oculus? Does that lower the value of a similar venture by the mall's owners?

Re: Real-time, in-camera background compositing in The Mandalorian

#65

Post by ishtanbul »

dd36 wrote: post #58
devindotcom wrote: post #5

This is super interesting stuff and I've been following it for some time. I just wrote it up with a bit more context: https://techcrunch.com/2020/02/20/how-the-mandalorian-and-il... It's not just ILM and Disney either, this is going to be everywhere . It's complex to set up and run in some ways but the benefits are enormous for pretty much everyone involved. I doubt there will be a major TV or film production that do…

I wonder how much this reduces the environmental footprint. The explosion in shows and movies always looking for raw nature or awesome settings has me wondering how much destruction it makes. And how much waste it produces.

Well maybe you arent trucking a huge crew out into the woods for weeks but you are creating a ton of disposable ewaste instead. Save on jet fuel too

Re: Real-time, in-camera background compositing in The Mandalorian

#66

Post by BubRoss »

dd36 wrote: post #58
devindotcom wrote: post #5

This is super interesting stuff and I've been following it for some time. I just wrote it up with a bit more context: https://techcrunch.com/2020/02/20/how-the-mandalorian-and-il... It's not just ILM and Disney either, this is going to be everywhere . It's complex to set up and run in some ways but the benefits are enormous for pretty much everyone involved. I doubt there will be a major TV or film production that do…

I wonder how much this reduces the environmental footprint. The explosion in shows and movies always looking for raw nature or awesome settings has me wondering how much destruction it makes. And how much waste it produces.

Environmental footprint? A local trade show, movie theater or small office building probably uses more electricity. Let's try to keep things in perspective.

Re: Real-time, in-camera background compositing in The Mandalorian

#67

Post by ishtanbul »

mdorazio wrote: post #31

For those wondering, this appears to be not nearly as expensive as I thought. The 20" square panels used are available for around $1000 each if you buy a lot of them used. Compared to a typical production budget for a high-quality franchise, it's surprisingly cheap to build one of these walls. The equipment to run it, on the other hand, is likely not cheap at all.

If the mandolorian was filmed entirely on location with vfx in post it would have cost hundreds and hundreds of millions. The sets were incredibly detailed. So i think they saved a ton of money for the output quality. I also doubt they bought second hand gear.

Re: Real-time, in-camera background compositing in The Mandalorian

#68

Post by BubRoss »

flashman wrote: post #29

I wonder how the photogrammetry aspect will intersect with intellectual property laws. The example used - scanning in a Santa Monica bar so that you can do reshoots without revisiting the location - would be an obvious example that might raise someone's hackles ("because it's our bar you're using to make your money" for instance). If you add that bar to your digital library, do you have to pay them royalties each tim…

When shooting at a location, the owner is payed a location fee. Detailed and specialized photography has been used at locations for decades at this point. This is a refinement of what is already happening, not something completely new.

Re: Real-time, in-camera background compositing in The Mandalorian

#69

Post by BubRoss »

ishtanbul wrote: post #67
mdorazio wrote: post #31

For those wondering, this appears to be not nearly as expensive as I thought. The 20" square panels used are available for around $1000 each if you buy a lot of them used. Compared to a typical production budget for a high-quality franchise, it's surprisingly cheap to build one of these walls. The equipment to run it, on the other hand, is likely not cheap at all.

If the mandolorian was filmed entirely on location with vfx in post it would have cost hundreds and hundreds of millions. The sets were incredibly detailed. So i think they saved a ton of money for the output quality. I also doubt they bought second hand gear.

Multiple hundreds of millions to shoot in the desert with red cameras and add buildings behind armored people with their hair hidden? Plenty of movies have been done like that and they didn't become the most expensive movies ever made (which is actually Pirates 3).

Re: Real-time, in-camera background compositing in The Mandalorian

#70

Post by re »

czr wrote: post #60
treblig wrote: post #59

"Postproduction was mostly refining creative choices that we were not able to finalize on the set in a way that we deemed photo-real." Does anyone know how they were able to swap out the in-camera version of the background originally shown on the LED wall with something more convincing later? Seems like it'd be tough since it's not a green screen!

iirc they project small green region around the actors and real props, so that ambient light and reflections are still mostly correct but they can also get clean matte out.

There's a demo of this here: https://youtu.be/Hjb-AqMD-a4?t=609
Post reply on HN