Real-time, in-camera background compositing in The Mandalorian
1–10 of 126 posts
Re: Real-time, in-camera background compositing in The Mandalorian
#2"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data."
Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking.
Re: Real-time, in-camera background compositing in The Mandalorian
#3Here is the Unreal Engine tech they are using: https://www.unrealengine.com/en-US/spotlights/unreal-engine-... . This is a video of it in action: https://www.youtube.com/watch?v=bErPsq5kPzE&feature=emb_logo .
Re: Real-time, in-camera background compositing in The Mandalorian
#4"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data." Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking. [0] https://www.youtube.com/watch?v=LC_KKxAuLQw
For the VFX industry, the tracking had already been solved for ages, with those reflective little balls on suits etc. in a mocap system.
The Wii sensor bar's thing was that it was really cheap.
But yes, damn close to a holodeck. But you can't see depth in this setup, right?
Re: Real-time, in-camera background compositing in The Mandalorian
#5This is super interesting stuff and I've been following it for some time. I just wrote it up with a bit more context:
https://techcrunch.com/2020/02/20/how-the-mandalorian-and-il...
It's not just ILM and Disney either, this is going to be everywhere. It's complex to set up and run in some ways but the benefits are enormous for pretty much everyone involved. I doubt there will be a major TV or film production that doesn't use LED walls 5 years from now.
Re: Real-time, in-camera background compositing in The Mandalorian
#6"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data." Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking. [0] https://www.youtube.com/watch?v=LC_KKxAuLQw
For the VFX industry, the tracking had already been solved for ages, with those reflective little balls on suits etc. in a mocap system. The Wii sensor bar's thing was that it was really cheap. But yes, damn close to a holodeck. But you can't see depth in this setup, right?
They have polarized 3D screens, and with head tracking, you have it.
Re: Real-time, in-camera background compositing in The Mandalorian
#7"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data." Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking. [0] https://www.youtube.com/watch?v=LC_KKxAuLQw
For the VFX industry, the tracking had already been solved for ages, with those reflective little balls on suits etc. in a mocap system. The Wii sensor bar's thing was that it was really cheap. But yes, damn close to a holodeck. But you can't see depth in this setup, right?
Mocap systems haven't really been able to produce deliverable results without human intervention very long. I'd argue they're still not there, some filtering and cleanup of the data is usually required. A lot of VFX is still about throwing human labour at problems.
Edit: I should note I'm talking about motion capture for characters etc. Capturing the motion of a rigid object like a camera in a controlled environment is very doable.
Re: Real-time, in-camera background compositing in The Mandalorian
#8Fascinating, but this article needs a proofread, damn...
"The virtual world on the LED screen is fantastic for many uses, but obviously an actor cannot walk through the screen, so an open doorway doesn't work when it's virtual. Doors are an aspect of production design that have to be physical. If a character walks through a door, it can’t be virtual, it must be real as the actor can’t walk through the LED screen."
Not to mention the multiple paragraphs that are basically re-stated immediately afterwards. It's like they hit publish in the middle of editing.
Re: Real-time, in-camera background compositing in The Mandalorian
#9The Mandalorian was probably a very likely candidate for this kind of approach, since it's essentially a western, meaning a lot of wide landscape shots.
The LED screen approach works nicely for fairly uncomplicated background geometry, like a plain. Try shooting Spiderman climbing up walls on that, and things will get tricky fast.
As the article notes, slow camera moves are a plus as well. The reason given is technical, but I also wonder how far you could really push the camera motion even if tracking lag wasn't an issue. The background is calculated to match the camera's viewpoint, so I expect it would be very disorienting for the actors if the camera was moving at high speeds.
Re: Real-time, in-camera background compositing in The Mandalorian
#10[deleted]