Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real-time, in-camera background compositing in The Mandalorian

ascmag.com

Post reply on HN
Print view View original

101–110 of 126 posts

Re: Real-time, in-camera background compositing in The Mandalorian

#101

Post by janekm »

treblig wrote: post #59

"Postproduction was mostly refining creative choices that we were not able to finalize on the set in a way that we deemed photo-real." Does anyone know how they were able to swap out the in-camera version of the background originally shown on the LED wall with something more convincing later? Seems like it'd be tough since it's not a green screen!

While currently they are using "green screen" in those instances, given that the camera positions are already being tracked, and the image displayed on the screens is known, it would be possible to re-render what image the camera should have seen if the foreground elements hadn't been present and use the difference with the recorded image as a mask for further post-processing.

(which would be very cool as it would also allow using a low-resolution version of the background during production that could then be re-rendered with a higher resolution image after the fact)

Re: Real-time, in-camera background compositing in The Mandalorian

#102

Post by werber »

mitchelhall wrote: post #50

Hi all, really cool that you have taken an interest in this project, a lot of your comments below are very insightful and interesting. I work on the team that deploys this tech on set. We focus on how the video signals get from the video output servers to the LED surfaces, coordinating how we deal with tracking methods, photogrammetry, signal delivery, operator controls and the infrastructure that supports all of thi…

That article is mindblowing, how do the tech side and the creative side work together on this kind of a project? How much does the technology shape the story telling?

Re: Real-time, in-camera background compositing in The Mandalorian

#103

Post by mnw21cam »

tobr wrote: post #73
devindotcom wrote: post #5

This is super interesting stuff and I've been following it for some time. I just wrote it up with a bit more context: https://techcrunch.com/2020/02/20/how-the-mandalorian-and-il... It's not just ILM and Disney either, this is going to be everywhere . It's complex to set up and run in some ways but the benefits are enormous for pretty much everyone involved. I doubt there will be a major TV or film production that do…

Is there any way to read your article if I can’t figure out how to navigate through the legalese and dark patterns of Verizon’s privacy extortion wall?

I don't see a privacy extortion wall, but I do see pictures with completely the wrong aspect ratio.

Re: Real-time, in-camera background compositing in The Mandalorian

#104

Post by jpmattia »

I've had a peripheral interest in virtual sets and real-time compositing by way of a colleague from grad school.

A quick visual summary of this tech: http://www.lightcrafttech.com/portfolio/beauty-beast/

This video was from a pilot several years ago, and it didn't make it to air, but it was visually wonderful.

Re: Real-time, in-camera background compositing in The Mandalorian

#105

Post by resoluteteeth »

This seems slightly limited in its current form in that they either have choose to either shoot the rendered background as-is (making it harder to modify in post production) or specifically turn the areas around the actors into green screens (sort of defeating the purpose).

I wonder if they could use some sort of trick like projectors synced with high fps cameras to make the real time rendering invisible to the cameras instead?

Re: Real-time, in-camera background compositing in The Mandalorian

#106

Post by mattigames »

vsareto wrote: post #20

Can you get a decent job just by knowing Unreal Engine well? Maybe by just doing small POC projects?

If by "well" you mean including Blueprints and physics-based-shaders then probably yes; although stuff like 3D rigging and modeling which is done on different third-party tools are a must for a lot related jobs.

Re: Real-time, in-camera background compositing in The Mandalorian

#107

Post by fhars »

Cthulhu_ wrote: post #82
miohtama wrote: post #16

Earlier quoted context omitted.

They mention they cannot push enough GPU juice to the screens, so they only render the camera focus area in full resolution. Also there is 12 frame lag which prevents moving camera too fast.

One solution to this would be to put the camera on a fixture that replays the same motions every time, so they can do a 'dry run' and correct the rendering. (Putting a camera on a fixture is not new, IIRC they did it in Back to the Future - https://www.youtube.com/watch?v=AtPA6nIBs5g is a really good yet succinct documentary on it)

IIRC they did it in Star Wars for the space battles (but it is forty years since I read about it, so may memory may be playing tricks on me).

Re: Real-time, in-camera background compositing in The Mandalorian

#108

Post by jweather »

marcan_42 wrote: post #92
kragen wrote: post #61

Earlier quoted context omitted.

Thanks! This work is really inspiring! Are these OLED screens, matrices of separate LEDs of the usual InGaAs and GaN type, or LCDs backlit with LEDs? The 2.84 mm pixel pitch makes it sound like it's separate inorganic LEDs. Are there times short of sunshine where you need more directionality to the lighting than the screens can provide? Because the screens can emit from any direction but not toward any direction, bei…

Discrete LED screens go all the way down to 2mm or even 1.9mm pitch (I have a few 2mm tiles at home). These are certainly discrete LED panels, not OLEDs. I'm not aware of OLEDs being used in "wall" applications like this. When you hear LED walls, think millions of discrete LEDs mounted on PCBs (and thank China for making this low cost enough to be viable!)

LED video walls are down in the 0.7mm dot pitch vicinity. Not quite the ~0.3mm of a 4K 55" LCD, but we're getting there. And the brightness and contrast ratio can't be beat.

Re: Real-time, in-camera background compositing in The Mandalorian

#109

Post by ygra »

fhars wrote: post #107
Cthulhu_ wrote: post #82

Earlier quoted context omitted.

One solution to this would be to put the camera on a fixture that replays the same motions every time, so they can do a 'dry run' and correct the rendering. (Putting a camera on a fixture is not new, IIRC they did it in Back to the Future - https://www.youtube.com/watch?v=AtPA6nIBs5g is a really good yet succinct documentary on it)

IIRC they did it in Star Wars for the space battles (but it is forty years since I read about it, so may memory may be playing tricks on me).

They did. They only had so many models of the spaceships, so multiple takes with a programmed camera path and compositing was used to increase the number of vessels in the scene.

Re: Real-time, in-camera background compositing in The Mandalorian

#110

Post by mercer »

overcast wrote: post #30
cbhl wrote: post #2

"The solution was ... a dynamic, real-time, photo-real background played back on a massive LED video wall and ceiling ... rendered with correct camera positional data." Gee, that sounds a lot like a holodeck. We've come a long way from using Wii Sensor Bars[0] for position tracking. [0] https://www.youtube.com/watch?v=LC_KKxAuLQw

Speaking of the Wii, does anyone else wish that motion controls could just be removed from its existing library? Not only has it just a major annoyance in most games, but its basically locked into that hardware now.

Metroid Prime is one of the few games I've played all the way through multiple times. When it became increasingly difficult to do so on a GameCube, I relented and tried the ii U version. Much as I've tried,I can't get comfortable with the controls.

Thankfully I finally have a laptop that can properly emulate the GC version and allow me to use the GC controller that I dusted off.

But I can't wait for a GC-inspired version on the Switch!

Post reply on HN