For those wondering, this appears to be not nearly as expensive as I thought. The 20" square panels used are available for around $1000 each if you buy a lot of them used. Compared to a typical production budget for a high-quality franchise, it's surprisingly cheap to build one of these walls. The equipment to run it, on the other hand, is likely not cheap at all.
Real-time, in-camera background compositing in The Mandalorian
31–40 of 126 posts
Re: Real-time, in-camera background compositing in The Mandalorian
#32Here is the Unreal Engine tech they are using: https://www.unrealengine.com/en-US/spotlights/unreal-engine-... . This is a video of it in action: https://www.youtube.com/watch?v=bErPsq5kPzE&feature=emb_logo .
That's really cool!
Re: Real-time, in-camera background compositing in The Mandalorian
#33So I guess the next step will just be movies made entirely inside of Virtual Reality environments, and an Actor in a MoCap suit who plays his virtual Avatar right?
Re: Real-time, in-camera background compositing in The Mandalorian
#34Earlier quoted context omitted.
If it's perspective corrected for the camera, it would probably look very distorted for anyone else on set -- whether there's depth or not And that's certainly not the goal with this. Something along those lines has been around for a while ( https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ... ). This system seems specifically targeted for solving challenges for film production, as it probably should be. I…
From the video posted upthread, around 3:40 it looks like they're doing just that: https://youtu.be/gUnxzVOs3rk?t=220
The most interesting aspect to me is that the system is pulling double duty by displaying both a dynamic, perspective-correct backdrop for the camera's POV and a static view for environment lighting and reflections outside of the camera's view frustum.
I wonder if they had to take care to mitigate artifacts caused by the dynamic view bouncing off of surfaces facing away from the camera.
Re: Real-time, in-camera background compositing in The Mandalorian
#35So we'll be able to play video game adaptations of the locations in the episodes really easily/soon, right? ;)
That has to be a consideration. But I don't know how much it would really speed up production of a AAA Mandalorian game. Some... maybe a 6 month head start on a 4-5 year game.
It would definitely help make the game environments higher quality and be a cost saving to the studio.
Re: Real-time, in-camera background compositing in The Mandalorian
#36"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data." Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking. [0] https://www.youtube.com/watch?v=LC_KKxAuLQw
The "holodeck" version of this is called a CAVE and the first one was built in 1992: https://www.youtube.com/watch?v=aKL0urEdtPU https://en.wikipedia.org/wiki/Cave_automatic_virtual_environ...
The haptics on CAVE were pretty clumsy, but boy howdy! I was pretty damn impressed when I got to interact with it at SIGGRAPH that following year.
I've got to say that the NC (Chapel Hill) pixel machine may have been a more supreme technical achievement, but it was harder to "get a feel for". As a young semiconductor geek at Motorola (Tokyo Design Ops) at the time, I did push for getting 1~4 bit CPUs on every column of 128-~192- bit deep framebuffers for a time. I almost got Sega to sponsor the project, but then something distracted me (I don't know what - but I suspect it was the Motorola 96K and a bank of AMD bit-slicer chips)) and I wandered off to do something else.
Re: Real-time, in-camera background compositing in The Mandalorian
#37I wonder how the photogrammetry aspect will intersect with intellectual property laws. The example used - scanning in a Santa Monica bar so that you can do reshoots without revisiting the location - would be an obvious example that might raise someone's hackles ("because it's our bar you're using to make your money" for instance). If you add that bar to your digital library, do you have to pay them royalties each tim…
A potential analogy might be something like using Carrie Fisher's image in the new Star Wars movies. I would assume the estate got paid for that. Or holo-tupac.
Practically speaking I think it will come down to what you negotiate. If you negotiate usage of the bar for your series then you can use it, otherwise not. If you negotiate resale of that model then that's legal, otherwise not. Most large productions will probably want to stay far on the right side of the law and get a written/financial agreement until things are hammered out, then you'll have amateur filmmakers who have to do vigilante shoots.
And again, probably something that will have to be legislated out for the long term.
In France, the appearance of buildings can be copyrighted, famously the Eiffel Tower is very aggressive about suing photographers.
Re: Real-time, in-camera background compositing in The Mandalorian
#38I wonder how the photogrammetry aspect will intersect with intellectual property laws. The example used - scanning in a Santa Monica bar so that you can do reshoots without revisiting the location - would be an obvious example that might raise someone's hackles ("because it's our bar you're using to make your money" for instance). If you add that bar to your digital library, do you have to pay them royalties each tim…
A potential analogy might be something like using Carrie Fisher's image in the new Star Wars movies. I would assume the estate got paid for that. Or holo-tupac. Practically speaking I think it will come down to what you negotiate. If you negotiate usage of the bar for your series then you can use it, otherwise not. If you negotiate resale of that model then that's legal, otherwise not. Most large productions will pro…
Royalties for use of the likeness of an actor was a fought for thing with the actor's guild after they did various tricky things to put Crispin Glover into Back to the Future 2 without his consent. I dunno that that really applies to locations in the same way.
Re: Real-time, in-camera background compositing in The Mandalorian
#39I wonder how the photogrammetry aspect will intersect with intellectual property laws. The example used - scanning in a Santa Monica bar so that you can do reshoots without revisiting the location - would be an obvious example that might raise someone's hackles ("because it's our bar you're using to make your money" for instance). If you add that bar to your digital library, do you have to pay them royalties each tim…
If you have to pay royalties, they won't be to the bar; they'll be to the bar's architect. Copyright law generally only covers expressive, rather than functional, aspects of a copyrighted work, so things like doors and walls might be okay, but architectural design is recognized as copyrightable.
In general I strongly recommend avoiding the term "intellectual property" because it conflates several different areas of law with almost totally unrelated traditions, statutes, and (in common-law countries) precedents — copyrights, patents, design patents, mask works, trademarks, trade secrets, and most alarmingly, in the EU, database rights. (Moreover, it's an extremely loaded term, like "pro-life" and "pro-choice".)
Re: Real-time, in-camera background compositing in The Mandalorian
#40This technique will produce potentially significant rendering artifacts in the final image. The backdrop is correct only from the position of the camera. A reflection from any surface will not be geometrically correct (as seen by the image from the article) I think that even ambient lighting would contain noticable deformations.
It's much better than the reflection of a green/blue screen or an empty studio with some cameras and people.
Glossy surfaces are usually not a problem unless they're (near) perfect mirrors. Even then -- lights are usually what you see in most reflections because they're orders of magnitude brighter than the rest of the set.
If reflections are problem with this new technique in certain settings, they would be even more with the current state of the art.
In those cases you replace them digitally. No way around this; either way.
Related trivia: The chrome starships in EP1 were actually rendered with environment mapping and reflection occlusion[1]. Even most games do better stuff today. Did you notice? :]
[1] http://www.spherevfx.com/downloads/ProductionReadyGI.pdf 5.3 Reflection Occlusion, pp. 89