Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stable Video Diffusion

stability.ai

Post reply on HN
Print view View original

111–120 of 316 posts

Re: Stable Video Diffusion

#111

Post by boppo1 »

epr wrote: post #104
TacticalCoder wrote: post #52

Earlier quoted context omitted.

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer…

> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.

Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run.

More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/my...

Re: Stable Video Diffusion

#112

Post by pessimizer »

r3d0c wrote: post #103
ben_w wrote: post #89

Earlier quoted context omitted.

Very far, yes, but also in a fast moving field. CGI in films used to be obvious all the time no matter how good the artists using it, now it's everywhere and only noticeable when that's the point; the gap from Tron to Fellowship of the Ring was 19.5 years. My guess is the analogy here puts the quality of existing genAI somewhere near the equivalent of early TV CGI, given its use in one of the Marvel title sequences e…

something unrelated improved overtime so something else unrelated will also improve to whatever goal you've set in your mind weird logic circles yall keep making to justify your beliefs, i mean the world is very easy like you just described if you completely strip all nuance and complexity people used to believe at the start of the space race we'd have mars colonies by now because they looked at the rate of technolog…

People also believed that recorded music would destroy the player piano industry and the market for piano rolls. Just because recorded music is cheaper doesn't mean that the audience will be willing to give up the actual sound of a piano being played.

Re: Stable Video Diffusion

#113

Post by boppo1 »

cptaj wrote: post #105
atentaten wrote: post #81

Earlier quoted context omitted.

Whats your reasoning for feeling that we're close?

We do it for text, audio and bitmapped images. A 3D scene file format is no different, you could train a model to output a blender file format instead of a bitmap. It can learn anything you have data for. Heck, we do it with geospatial data already, generating segmentation vectors. Why not 3D?

>3D scene file format is no different

Not in theory, but the level of complexity is way higher and the amount of data available is much smaller.

Compare bitmaps to this: https://fossies.org/linux/blender/doc/blender_file_format/my...

Re: Stable Video Diffusion

#114

Post by hanniabu »

valine wrote: post #7

The rate of progress in ML this past year has been breath taking. I can’t wait to see what people do with this once controlnet is properly adapted to video. Generating videos from scratch is cool, but the real utility of this will be the temporal consistency. Getting stable video out of stable diffusion typically involves lots of manual post processing to remove flicker.

> but the real utility of this will be the temporal consistency

The main utility will me misinformation

Re: Stable Video Diffusion

#115

Post by alberth »

mlboss wrote: post #109
alberth wrote: post #107

Earlier quoted context omitted.

What was the big “unlock” that allowed so much progress this past year? I ask as a noob in this area.

Stable diffusion open source release and llama release

But what technically allowed for so much progress?

There’s been open source AI/ML for 20+ years.

Nothing comes close to the massive milestones over the past year.

Re: Stable Video Diffusion

#116

Post by 4death4 »

alberth wrote: post #107
valine wrote: post #7

The rate of progress in ML this past year has been breath taking. I can’t wait to see what people do with this once controlnet is properly adapted to video. Generating videos from scratch is cool, but the real utility of this will be the temporal consistency. Getting stable video out of stable diffusion typically involves lots of manual post processing to remove flicker.

What was the big “unlock” that allowed so much progress this past year? I ask as a noob in this area.

I think these are the main drivers behind the progress:

- Unsupervised learning techniques, e.g. transformers and diffusion models. You need unsupervised techniques in order to utilize enough data. There have been other unsupervised techniques in the past, e.g. GANs, but they don't work as well.

- Massive amounts of training data.

- The belief that training these models will produce something valuable. It costs between hundreds of thousands to millions of dollars to train these models. The people doing the training need to believe they're going to get something interesting out at the end. More and more people and teams are starting to see training a large model as something worth pursuing.

- Better GPUs, which enables training larger models.

- Honestly the fall of crypto probably also contributed, because miners were eating a lot of GPU time.

Re: Stable Video Diffusion

#117

Post by ben_w »

r3d0c wrote: post #103
ben_w wrote: post #89

Earlier quoted context omitted.

Very far, yes, but also in a fast moving field. CGI in films used to be obvious all the time no matter how good the artists using it, now it's everywhere and only noticeable when that's the point; the gap from Tron to Fellowship of the Ring was 19.5 years. My guess is the analogy here puts the quality of existing genAI somewhere near the equivalent of early TV CGI, given its use in one of the Marvel title sequences e…

something unrelated improved overtime so something else unrelated will also improve to whatever goal you've set in your mind weird logic circles yall keep making to justify your beliefs, i mean the world is very easy like you just described if you completely strip all nuance and complexity people used to believe at the start of the space race we'd have mars colonies by now because they looked at the rate of technolog…

First, lotta artists already upset with genAI and the impact it has.

Second, I literally wrote the same point you seem to think is a gotcha:

> it is just an analogy and there's no guarantees of anything either way

Re: Stable Video Diffusion

#118

Post by mikepurvis »

boppo1 wrote: post #111
epr wrote: post #104

Earlier quoted context omitted.

> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.

Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run. More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/my...

But surely you wouldn't try to emit that format directly, but rather some higher level scene description? Or even just a set of instructions for how to manipulate the UI to create the imagined scene?

Re: Stable Video Diffusion

#120

Post by Cyphase »

alberth wrote: post #107
valine wrote: post #7

The rate of progress in ML this past year has been breath taking. I can’t wait to see what people do with this once controlnet is properly adapted to video. Generating videos from scratch is cool, but the real utility of this will be the temporal consistency. Getting stable video out of stable diffusion typically involves lots of manual post processing to remove flicker.

What was the big “unlock” that allowed so much progress this past year? I ask as a noob in this area.

One factor is that Stable Diffusion and ChatGPT were released within 3 months of each other – August 22, 2022 and November 3, 2022, respectively. That brought a lot of attention and excitement to the field. More excitement, more people, more work being done, more progress.

Of course those two releases didn't fall out of the sky.

Post reply on HN