Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

151–160 of 430 posts

Re: Diffusion models are real-time game engines

#151

Post by troupo »

reverius42 wrote: post #121
troupo wrote: post #86

Earlier quoted context omitted.

> one can imagine a near future where text prompts are enough to create a fun new game Sit down and write down a text prompt for a "fun new game". You can start with something relatively simple like a Mario-like platformer. By page 300, when you're about halfway through describing what you mean, you might understand why this is wishful thinking

If it can be trained on (many) existing games, then it might work similarly to how you don't need to describe every possible detail of a generated image in order to get something that looks like what you're asking for (and looks like a plausible image for the underspecified parts).

Things that might work plausible in a static image will not look plausible when things are moving, especially in the game.

Also: https://news.ycombinator.com/item?id=41376722

Also: define "fun" and "new" in a "simple text prompt". Current image generators suck at properly reflecting what you want exactly, because they regurgitate existing things and styles.

Re: Diffusion models are real-time game engines

#152

Post by lupusreal »

lIl-IIIl wrote: post #106

How does it know how many times it needs to shoot the zombie before it dies? Most enemies have enough hit points to survive the first shot. If the model is only trained on the previous frame, it doesn't know how many times the enemy was already shot at. From the video it seems like it is probability based - they may die right away or it might take way longer than it should. I love how the player's health goes down wh…

> In Doom the enemies fight with each other if they accidentally incur "friendly fire". It would be interesting to see it play out in this version.

They trained this thing on bot gameplay, so I bet it does poorly when advanced strategies like deliberately inducing mob infighting are employed (the bots probably didn't do that a lot, of at all.)

Re: Diffusion models are real-time game engines

#153

Post by soulofmischief »

golol wrote: post #147
Closi wrote: post #109

Earlier quoted context omitted.

Because video is much more difficult than images (it's lots of images that have to be consistent across time, with motion following laws of physics etc), and this is much more limited in terms of scope than pure arbitrary video generation.

This misses the point, I'm comparing two methods of generating minecraft videos.

By simplifying the problem, we are better able to focus on researching specific aspects of generation. In this case, they synthetically created a large, highly domain-specific training set and then used this to train a diffusion model which encodes input parameters instead of text.

Sora was trained on a much more diverse dataset, and so has to learn more general solutions in order to maintain consistency, which is harder. The low resolution and simple, highly repetitive textures of doom definitely help as well.

In general, this is just an easier problem to approach because of the more focused constraints. It's also worth mentioning that noise was added during the process in order to make the model robust to small perturbations.

Re: Diffusion models are real-time game engines

#154

Post by csmattryder »

Jensson wrote: post #89
arduinomancer wrote: post #42

How does the model “remember” the whole state of the world? Like if I kill an enemy in some room and walk all the way across the map and come back, would the body still be there?

It doesn't even remember the state of the game you look at. Doors spawning right in front of you, particle effects turning into enemies mid flight etc, so just regular gen AI issues. Edit: Can see this in the first 10 seconds of the first video under "Full Gameplay Videos", stairs turning to corridor turning to closed door for no reason without looking away.

There's also the case in the video (0:59) where the player jumps into the poison but doesn't take damage for a few seconds then takes two doses back-to-back - they should've taken a hit of damage every ~500-1000ms(?)

Guessing the model hasn't been taught enough about that, because most people don't jump into hazards.

Re: Diffusion models are real-time game engines

#157

Post by icoder »

marci wrote: post #131
icoder wrote: post #119

This is impressive. But at the same time, it can't count. We see this every time, and I understand why it happens, but it is still intriguing. We are so close or in some ways even way beyond, and yet at the same time so extremely far away, from 'our' intelligence. (I say it can't count because there are numerous examples where the bullet count glitches, it goes right impressively often, but still, counting, being up…

'our' intelligence may not be the best thing we can make. It would be like trying to only make planes that flaps wings or trucks with legs. A bit like using a llm to do multiplication. Not the best tool. Biomimcry is great for inspiration, but shouldn't be a 1-to-1 copy, especialy in different scale and medium.

Sure, although I still think a system with less of a contrast between how well it performs 'modally' and how bad it performs incidentally, would be more practical.

What I wonder is whether LLM's will inherently always have this dichotomy and we need something 'extra' (reasoning, attention or something les biomimicried), or whether this will eventually resolves itself (to an acceptable extend) when they improve even further.

Re: Diffusion models are real-time game engines

#160

Post by snickmy »

godelski wrote: post #137
snickmy wrote: post #129

Earlier quoted context omitted.

Those are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequen…

I'm not sure what you're saying is irrelevant. 1) the model has enough memory to store not only all game assets and engine but even hundreds of "plays". 2) me mentioning that there's still a lot of room to make these things better (seems you think so too so maybe not this one?) 3) an interesting point I was wondering to compare current state of things (I mean I'll give you this but it's just a random thought and I'm…

Sorry, probably didn't explain myself well enough

1) yes you are correct. the point i was making is that, in the context of the discovery/research, that's outside the scope, and 'easier' to do, as it has been done in other verticals (ie.: e2e self driving)

2) yep, aligned here

3) I'm not fully following here, but agree this is not NeurIPS, and no Schmidhuber's bickering.

4) The network does store information, it just doesn't store a gameplay information, which could be forced, but as per point 1, it is , and I think it is the right approach, beyond the scope of this research

Post reply on HN