Live data from Hacker News

SANA-WM, a 2.6B open-source world model for 1-minute 720p video

nvlabs.github.io

41–50 of 162 posts

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#41

Earlier quoted context omitted.

Games. Build campaigns in hours instead of months. Make it possible for users to create their own campaigns, move the action to different game worlds - 'gimme Mario Kart in the ${favourite_game} world', etc.

Yeah, but is this really that great? Are these models going to remember the town you wandered through on your session yesterday and want to return to? Imagine playing Read Dead Redemption 2 and you attempt to ride your horse from Saint Denis to Valentine and Valentine no longer exists, or is a completely different town located half a mile off from where it was originally. I just don't see how this would work...

Yes, a lot of models don’t state this explicitly, but they can be made deterministic. Not the generation itself, but the same prompt, with a generation seed will always result in the same output.

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#42
post #16

I struggle with these world models from the perspective of video games (so this post is a particular perspective). I'm not a game developer myself, but some of my favorite games carry a deep sense of intentionality. For instance, there is typically not a single item misplaced in a FromSoftware game (or, for instance, Lies of P -- more recently). Almost every object is placed intentionally. Games which lack this inten…

There are two things here, firstly, Without AI, you can have heavily designed environments or you can have procedurally generated, people manage to make both work. Both can also fail because of reasons specific to the approach. Careless procedural generation can produce a poor variety or nonsensical outputs. Careless specific placement can violate any rules that a game has established creating an incoherent experience.

Making a world internally consistent by explicit placement gets harder as you increase in scale. When internal consistency is a factor impacting quality, there is a scale at which generated content eventually becomes the higher quality solution.

Secondly, when generating content with AI, the same rules around carelessness apply. There are certainly generative AI tools out there that offer few options when it comes to composing what you want, that is not a necessary part of AI, some of it is because people are wanting rudimentary interfaces, some of it is that the generators are sufficiently new that the control mechanisms are limited because they are focused upon doing something at all before doing it highly controlled, in some ways the problem is that things are new enough that it can be hard to describe what is desirable controllability, making the generator to see what people would like it to be able to do is, I think, a reasonable path to follow prior to creating the control that people want. Part of it is also that there _are_ tools that give a high level of control over what is generated but far fewer people get to see them. There are ways to control styles, object placement, camera motions, scene compositions, etc. The more specialised you get, the smaller the subset of people who need that specific control.

I think AI can make things possible for people who could not have done so without them, but it's still going to take care to make something special.

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#44

What’s the long term utility of world models? There’s no doubt they’re technically impressive, but what does one do with it?

I believe the idea is to offer simulation of ideas to test out new tasks AND something like dreaming.

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#45

Earlier quoted context omitted.

By and large I agree, but it doesn’t need to be either/or. Many of the most popular games in the past decade are procedurally generated and have nothing “intentionally” placed (apart from tuning/tweaking the balance of the seeding algorithms).

Which game would that be apart from Minecraft?

No Man's Sky, Terraria, Dead Cells, to name a few.

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#46
post #16

I struggle with these world models from the perspective of video games (so this post is a particular perspective). I'm not a game developer myself, but some of my favorite games carry a deep sense of intentionality. For instance, there is typically not a single item misplaced in a FromSoftware game (or, for instance, Lies of P -- more recently). Almost every object is placed intentionally. Games which lack this inten…

Video games are not the initial motivation at all.

These world models are key for robotic and coherence in video generation.

Give a world model images of a factory, the robot now can simulate tasks and do the best result.

Give a world model images/context etc. and it can generate a coherent world for video generation.

What this world model system might be able to do for us in regards of gaming or virtual reality: Either simulate 'old' environments like the house of your grandparents (gaussian splatting but interactive) or potential new ones like a house, kitchen, remodeling.

It can also be a very interesting easy to approach VR environment were you can start building your world with voice. That would be very intentional. After all world building is not necessarily connected to being able to generate 3d assets. Just because you need to go this route today, doesn't mean you have to do this tomorrow.

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#47

What’s the long term utility of world models? There’s no doubt they’re technically impressive, but what does one do with it?

The world model is useful for planning. It can "anticipate" consequences of actions. This can be used for a kind of tree search to decide on optimal actions in robotics

Re: SANA-WM, a 2.6B open-source world model for 1-minute 720p video

#49

Earlier quoted context omitted.

By and large I agree, but it doesn’t need to be either/or. Many of the most popular games in the past decade are procedurally generated and have nothing “intentionally” placed (apart from tuning/tweaking the balance of the seeding algorithms).

> have nothing “intentionally” placed (apart from tuning/tweaking the balance of the seeding algorithms). I think you underestimate the intentionality that goes into developing procedural generation. Something like Dwarf Fortress isn't "place objects randomly" - it is layers upon layers of carefully crafted systems that build upon each other to produce specific patterns of outcome

By calling it out in my comment, I was trying to not underestimate it.

I guess what I'm saying is: Couldn't a world model with targeted training and thoughtfully tuned system prompts be directionally similar to the layered systems to produce specific patterns of outcome?

Post reply on HN