Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiment: Can 3D improve AI video consistency?

backdroptech.github.io

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Experiment: Can 3D improve AI video consistency?

#31

Post by fabiopolimeni »

weinzierl wrote: post #26

It would be surprising if it didn't, but it's a question of diminishing returns. The complexity and cost likely increase approximately with the third power. Does the result justify this effort?"

The rationale is when you do not use 3D models to keep subjects consistent over multiple scenes it becomes too unpredictable and you would need way too many trial and errors. 3D helps with it. Then, if you even want to have that very same character starting from a specific pose it will more likely you will give up to the likes of the AI model rather than the other way around.

Re: Experiment: Can 3D improve AI video consistency?

#32

Post by master-lincoln »

TobiasEnholmX wrote: post #6
daef wrote: post #3

i am very confused... what is 3D? to me 3D means three dimensions - but it looks like 3D refers to a product? this page doesn't help with explaining anything

Good question! When we say '3D' here, we mean using 3D models and scenes as a base to guide AI video generation. The AI isn’t just making things up from scratch—it follows the structure of a 3D-rendered scene to keep things consistent across frames. It’s not a product, just an approach to solving the problem of AI-generated video drifting too much between frames.

> 3D models and scenes

Maybe add "digital" too...

Re: Experiment: Can 3D improve AI video consistency?

#34

Post by smusamashah »

Article doesn't clearly tell what its doing.

Is it training a model based on 3d model? Is it doing img2img? is doing video2video using 3d video/image as source?

Article is showing off stuff but is light on explanation of what is going on here.

Also, the person is bearded. Hard to notice face changes there. I want to see more demos but instead of a 3d like render, do a realistic render without beard etc.

edit: Plus these are very short clips. 3D should theoretically help, but this article could have been better.

edit2: in the anime example, the girls face is definitely changing (+lips are not correct). It feels like you are doing img2video here?

Re: Experiment: Can 3D improve AI video consistency?

#35

Post by TobiasEnholmX »

smusamashah wrote: post #34

Article doesn't clearly tell what its doing. Is it training a model based on 3d model? Is it doing img2img? is doing video2video using 3d video/image as source? Article is showing off stuff but is light on explanation of what is going on here. Also, the person is bearded. Hard to notice face changes there. I want to see more demos but instead of a 3d like render, do a realistic render without beard etc. edit: Plus th…

Appreciate the feedback! Sounds like you're looking for more of a deep dive into the tech and how it works. This post was more about showing results, but we’ll cover the process and technical details in another one

Re: Experiment: Can 3D improve AI video consistency?

#36

Post by karmakaze »

TobiasEnholmX wrote: post #2

AI-generated video struggles with consistency. Flickering, weird proportions, and characters changing. I tried using 3D as a way to get more consistency. Overall, it worked. No sudden changes in proportions, clothing, or style. Still, there are some limitations, especially with fine details. We’re looking into whether this could be useful as a tool and would love to hear what you think: Has anyone experimented with 3…

Nice demo examples. I'm a casual observer and curious how this differs from Gaussian splatting which also (implicitly?) uses 3D representations.

I could see applying changes at the 3D model level which wouldn't be directly accessible if it was only an internal representation.

Re: Experiment: Can 3D improve AI video consistency?

#38

Post by TobiasEnholmX »

karmakaze wrote: post #36
TobiasEnholmX wrote: post #2

AI-generated video struggles with consistency. Flickering, weird proportions, and characters changing. I tried using 3D as a way to get more consistency. Overall, it worked. No sudden changes in proportions, clothing, or style. Still, there are some limitations, especially with fine details. We’re looking into whether this could be useful as a tool and would love to hear what you think: Has anyone experimented with 3…

Nice demo examples. I'm a casual observer and curious how this differs from Gaussian splatting which also (implicitly?) uses 3D representations. I could see applying changes at the 3D model level which wouldn't be directly accessible if it was only an internal representation.

Yeah, exactly. Gaussian Splatting works great when you have an image (or set of images) and want to reconstruct a whole scene in 3D, but it treats everything as a unified point-based representation. In a structured 3D scene, though, objects are clearly separated, so you can manipulate them individually.

For example, you can attach a LoRA specifically to one object and run a separate workflow just for that, giving you way more control. That’s a big difference—Gaussian Splatting doesn’t naturally lend itself to object-level edits since everything is blended into the same representation.

Post reply on HN