Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Experiment: Can 3D improve AI video consistency?

backdroptech.github.io

Post reply on HN
Print view View original

1–10 of 38 posts

Re: Experiment: Can 3D improve AI video consistency?

#2

Post by TobiasEnholmX »

AI-generated video struggles with consistency. Flickering, weird proportions, and characters changing. I tried using 3D as a way to get more consistency.

Overall, it worked. No sudden changes in proportions, clothing, or style. Still, there are some limitations, especially with fine details.

We’re looking into whether this could be useful as a tool and would love to hear what you think: Has anyone experimented with 3D + AI generation for images or video? or sees a better way to approach this?

Demo and details in the blog: https://backdroptech.github.io/3d-to-video/

Re: Experiment: Can 3D improve AI video consistency?

#4

Post by tomw1808 »

Is the idea here to basically transform the almost unlimited degrees of freedom an AI has in manipulating rastered image properties from one frame to another in a limited set of known things it can change?

I mean, unless its a wild magic sci-fi movie, it makes sense that once a character is known in separation to its background, that the background doesn't change with a hands movement of the character. And the character cannot move beyond earthy physics?

Is that what this is?

Re: Experiment: Can 3D improve AI video consistency?

#5

Post by TobiasEnholmX »

tomw1808 wrote: post #4

Is the idea here to basically transform the almost unlimited degrees of freedom an AI has in manipulating rastered image properties from one frame to another in a limited set of known things it can change? I mean, unless its a wild magic sci-fi movie, it makes sense that once a character is known in separation to its background, that the background doesn't change with a hands movement of the character. And the charac…

Yeah, that’s pretty much the idea. AI has a lot of freedom when generating each frame independently, which often leads to inconsistencies. By using a 3D base, we’re constraining that freedom. Locking in composition, lighting, and perspective so AI doesn’t hallucinate unintended changes. The goal is to have a stable structure where AI can still add detail and style but without breaking spatial or physical consistency.

Re: Experiment: Can 3D improve AI video consistency?

#6

Post by TobiasEnholmX »

daef wrote: post #3

i am very confused... what is 3D? to me 3D means three dimensions - but it looks like 3D refers to a product? this page doesn't help with explaining anything

Good question! When we say '3D' here, we mean using 3D models and scenes as a base to guide AI video generation. The AI isn’t just making things up from scratch—it follows the structure of a 3D-rendered scene to keep things consistent across frames. It’s not a product, just an approach to solving the problem of AI-generated video drifting too much between frames.

Re: Experiment: Can 3D improve AI video consistency?

#7

Post by ManuelKiessling »

I'm aware this is only a very tangential comparison, but my LLM-based coding workflows around Cursor regularly involve me creating "empty" skeletons of files which I add to the context, as in: because I'm aware that it makes sense for an implementation to have a FooBarService and two entities Fizz and Buzz, I create these as empty files (with the "class FoobarService" line as it's only content), and this way, Cursor (or rather, Claude) doesn't get too creative and wild when deciding how and where to implement code.

This really increased the quality of results for me.

Re: Experiment: Can 3D improve AI video consistency?

#10

Post by SebaSeba »

Not an expert here. I am not sure how 3D the videos in the article are. IMO they are 3D in a pixar/animated sort of way.

But I have very recent first hand experience of creating a video for our startup's Facebook post with Minimax image-to-video inference, from an image of our animated avatar character.

...And yes, first the videos were bad quality with lots of inconsistencies, but after adding "animated" to the prompt, in front of the "man" word, the result was pretty great already on the first try! Which I then ended up even using. (you can even check it here if interested https://fb.watch/xRC-fptexM/)

Perhaps it should be self-evident, but still, it was not to me. :)

Edit. I guess my point was also that the animated character in the video ended up being somewhat 3D as well.

Post reply on HN