Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stable Video Diffusion

stability.ai

Post reply on HN
Print view View original

201–210 of 316 posts

Re: Stable Video Diffusion

#201

Post by jncfhnb »

dragonwriter wrote: post #177
jncfhnb wrote: post #173

Earlier quoted context omitted.

Text, audio, and bitmapped images are data. Numbers and tokens. A 3D scene is vastly more complex, and the way you consume it is tangential to the rendering of it we use to interpret. It is a collection of arbitrary data structures. We’ll need a new approach for this kind of problem

> Text, audio, and bitmapped images are data. Numbers and tokens. > A 3D scene is vastly more complex 3D scenes, in fact, are also data, numbers and tokens. (Well, numbers, but so are tokens.)

As I stated and you selectively omitted, 3D scenes are collections of many arbitrary data structures.

Not at all the same as fixed sized arrays representing images.

Re: Stable Video Diffusion

#202

Post by dragonwriter »

jncfhnb wrote: post #201
dragonwriter wrote: post #177

Earlier quoted context omitted.

> Text, audio, and bitmapped images are data. Numbers and tokens. > A 3D scene is vastly more complex 3D scenes, in fact, are also data, numbers and tokens. (Well, numbers, but so are tokens.)

As I stated and you selectively omitted, 3D scenes are collections of many arbitrary data structures. Not at all the same as fixed sized arrays representing images.

Text gen, one of the things you contrast 3d to, similarly isn't fixed size (capped in most models, but not fixed.)

In fact, the data structures of a 3D scene can be serialized as text, and a properly trained text gen system could generate such a representation directly, though that's probably not the best route to decent text-to-3d.

Re: Stable Video Diffusion

#203

Post by solarkraft »

TacticalCoder wrote: post #52
btbuildem wrote: post #11

In the video towards the bottom of the page, there are two birds (blue jays), but in the background there are two identical buildings (which look a lot like the CN Tower). CN Tower is the main landmark of Toronto, whose baseball team happens to be the Blue Jays. It's located near the main sportsball stadium downtown. I vaguely understand how text-to-image works, and so it makes sense that the vector space for "blue j…

> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer…

Where is the training data coming from?

Re: Stable Video Diffusion

#204

Post by 01100011 »

btbuildem wrote: post #11

In the video towards the bottom of the page, there are two birds (blue jays), but in the background there are two identical buildings (which look a lot like the CN Tower). CN Tower is the main landmark of Toronto, whose baseball team happens to be the Blue Jays. It's located near the main sportsball stadium downtown. I vaguely understand how text-to-image works, and so it makes sense that the vector space for "blue j…

I recently tried to generate clip art for a presentation using GPT-4/DALL-E 3. I found it could handle some updates but the output generally varied wildly as I tried to refine the image. For instance, I'd have a cartoon character checking its watch and also wearing a pocket watch. Trying to remove the pocket watch resulted in an entirely new cartoon with little stylistic continuity to the first.

Also, I originally tried to get the 3 characters in the image to be generated simultaneously, but eventually gave up as DALL-E had a hard time understanding how I wanted them positioned relative to each other. I just generated 3 separate characters and positioned them in the same image using Gimp.

Re: Stable Video Diffusion

#206

Post by jdaxe »

KineticLensman wrote: post #68
spaceman_2020 wrote: post #29

A seemingly off topic question, but with enough compute and optimization, could you eventually simulate “reality”? Like, at this point, what are the technical counters to the assertion that our world is a simulation?

(disclaimer: worked in the sim industry for 25 years, still active in terms of physics-based rendering). First off, there are zero technical proofs that we are in a sim, just a number of philosophical arguments. In practical terms, we cannot yet simulate a single human cell at the molecular level, given the massive number of interactions that occur every microsecond. Simulating our entire universe is not technically…

Maybe something like quantum mechanics are an "optimization" of the sim, i.e the sim doesn't actually compute the locations, spin etc of subatomic particles but instead just uses probabilities to simulate it. Only when a consciousness decides to look more closely does it retroactively decide what those properties really were.

Kind of like how video games won't render the full resolution textures when the character is far away or zoomed out.

I'm sure I'm not the first person to have thought this.

Re: Stable Video Diffusion

#207

Post by p1esk »

sterlind wrote: post #165
cptaj wrote: post #101

Earlier quoted context omitted.

Probably not. But there does seem to be a clear path to it. The main issue is going to be having the right dataset. You basically need to record user actions in something like blender (ie: moving a model of a bike to the left of a scene), match it to a text description of the action (ie; "move bike to the left") and match those to before/after snapshots of the resulting file format. You need a whole metric fuckton of…

The hard part of AGI is the self-training and few examples. Your parents didn't attach strings to your body and puppeteer you through a few hundred thousand games of baseball. And the humans that invented baseball had zero training data to go on.

Your body is a result of a billion year old evolutionary optimization process. GPT-4 was trained from scratch in a few months.

Re: Stable Video Diffusion

#208

Post by mschuster91 »

alberth wrote: post #115
mlboss wrote: post #109

Earlier quoted context omitted.

Stable diffusion open source release and llama release

But what technically allowed for so much progress? There’s been open source AI/ML for 20+ years. Nothing comes close to the massive milestones over the past year.

> But what technically allowed for so much progress?

The availability of GPU compute time. Up until the Russian invasion into Ukraine, interest rates were low AF so everyone and their dog thought it would be a cool idea to mine one or another sort of shitcoin. Once rising interest rates killed that business model for good, miners dumped their GPUs on the open market, and an awful lot of cloud computing capacity suddenly went free.

Re: Stable Video Diffusion

#209

Post by firefoxd »

I understand the magnitude of innovation that's going on here. But still feel like we are generating these videos with both hands tied behind our backs. In other words, it's nearly impossible to edit the videos in this constraints. (Imagine trying to edit the blue Jays to get the perfect view).

Since videos are rarely consumed raw, what if this becomes a pipeline in Blender instead? (Blender the 3d software). Now the video becomes a complete scene with all the key elements of the text input animated. You have your textures, you have your animation, you have your camera, you have all the objects in place. We can even have the render engine in the pipeline to increase the speed of video generation.

It may sound like I'm complaining, but I'm just ask making a feature request...

Re: Stable Video Diffusion

#210

Post by stevage »

btbuildem wrote: post #11

In the video towards the bottom of the page, there are two birds (blue jays), but in the background there are two identical buildings (which look a lot like the CN Tower). CN Tower is the main landmark of Toronto, whose baseball team happens to be the Blue Jays. It's located near the main sportsball stadium downtown. I vaguely understand how text-to-image works, and so it makes sense that the vector space for "blue j…

I wondered similarly whether the astronaut's weird gait was because it was kind of "moonwalking" on the moon.
Post reply on HN