Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stable Video Diffusion

stability.ai

Post reply on HN
Print view View original

191–200 of 316 posts

Re: Stable Video Diffusion

#191

Post by filipezf »

cptaj wrote: post #101
p1esk wrote: post #69

Earlier quoted context omitted.

Are you working on all that?

Probably not. But there does seem to be a clear path to it. The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format. You need a whole metric fuckton of…

I have for some time planning to do a 'Wikipedia for AI' (even bought a domain), where people could contribute all sorts of these skills ( not only 3d video, but also manual skills, or anything). Given the current climate of 'AI will save/doom us', and that users would in some sense be training their own replacements, I don't know how much love such site would have, though.

Re: Stable Video Diffusion

#192

Post by adventured »

capableweb wrote: post #91
Der_Einzige wrote: post #37

Earlier quoted context omitted.

Controlnet is adapted to video today, the issues are that it's very slow. Haven't you seen the insane quality of videos on civitai?

> Haven't you seen the insane quality of videos on civitai? I have not, so I went to https://civitai.com/ which I guess is what you're talking about? But I cannot find a single video there, just images and models.

https://civitai.com/images

Go there, in the top right of the content area it has two drop-downs: Most Reactions | Filters

Under filters, change the media setting to video.

Civitai has a notoriously poor layout for finding/browsing things unfortunately.

Re: Stable Video Diffusion

#193

Post by n2d4 »

ebalit wrote: post #147
mkaic wrote: post #130

Earlier quoted context omitted.

I don't think transformers or diffusion models are inherently "unsupervised", especially not the way they're used in Stable Diffusion and related models (which are very much trained in a supervised fashion). I agree with the rest of your points though.

Generative methods have usually been considered unsupervised. You're right that conditional generation start to blur the lines though.

"Generative AI" is a misnomer; it's not the same kind of "generative" as the G in GAN.

While you're right about GANs, diffusion models as transformers as transformers are most commonly trained with supervised learning.

Re: Stable Video Diffusion

#194

Post by ebalit »

n2d4 wrote: post #193
ebalit wrote: post #147

Earlier quoted context omitted.

Generative methods have usually been considered unsupervised. You're right that conditional generation start to blur the lines though.

"Generative AI" is a misnomer; it's not the same kind of "generative" as the G in GAN. While you're right about GANs, diffusion models as transformers as transformers are most commonly trained with supervised learning.

I disagree. Diffusion models are trained to generate the probability distribution of their training dataset, like other generative models (GAN, VAE, etc). The fact that the architecture is a Transformer (or a CNN with attention like in Stable Diffusion) is orthogonal to the generative vs discriminative divide.

Unsupervised is a confusing term as there is always an underlying loss being optimized and working as a supervision signal, even for good old kmeans. But generative models are generally considered to be part of unsupervised methods.

Re: Stable Video Diffusion

#195

Post by insanitybit »

epr wrote: post #104
TacticalCoder wrote: post #52

Earlier quoted context omitted.

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer…

> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.

I assume because it's still extremely early.

Re: Stable Video Diffusion

#196

Post by [deleted] »

boppo1 wrote: post #111
epr wrote: post #104

Earlier quoted context omitted.

> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.

Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run. More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/my...

[deleted]

Re: Stable Video Diffusion

#197

Post by taberiand »

SXX wrote: post #157
ericpauley wrote: post #8

I'm still puzzled as to how these "non-commercial" model licenses are supposed to be enforceable. Software licenses govern the redistribution of the software , not products produced with it. An image isn't GPL'd because it was produced with GIMP.

It doesn't have to be enforceable. This licensing model works exactly the same as Microsoft Windows licensing or WinRAR licensing. Lots and lots of people have pirated Windows or just buy some cheap keys off Ebay, but no one of them in their sane mind would use anything like that at their company. The same way you can easily violate any "non-commercial" clauses of models like this one as private person or as some tin…

I've heard companies also intentionally do not go after individuals pirating software e.g., Adobe Photoshop - it benefits them to have students pirate and skill up on their software and then enter companies that buy Photoshop because their employees know it, over locking down and having those students, and then the businesses, switch to open source.

Re: Stable Video Diffusion

#198

Post by kornesh »

valine wrote: post #7

The rate of progress in ML this past year has been breath taking. I can’t wait to see what people do with this once controlnet is properly adapted to video. Generating videos from scratch is cool, but the real utility of this will be the temporal consistency. Getting stable video out of stable diffusion typically involves lots of manual post processing to remove flicker.

Yeah, solving the flickering problem and achieving temporal consistency will be the key to realize the full potential of generative video models.

Right now, AnimateDiff is leading the way in consistency but I'm really excited to see what people will do with this new model.

Re: Stable Video Diffusion

#199

Post by throwaway290 »

alberth wrote: post #107
valine wrote: post #7

The rate of progress in ML this past year has been breath taking. I can’t wait to see what people do with this once controlnet is properly adapted to video. Generating videos from scratch is cool, but the real utility of this will be the temporal consistency. Getting stable video out of stable diffusion typically involves lots of manual post processing to remove flicker.

What was the big “unlock” that allowed so much progress this past year? I ask as a noob in this area.

MS subsidizing it with 10 billions USD and (un)healthy contempt towards copyright.
Post reply on HN