Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

401–410 of 430 posts

Re: Diffusion models are real-time game engines

#401

Post by Philpax »

bangaladore wrote: post #383
wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

> It's insane that that this works, and that it works fast enough to render at 20 fps. It is running on an entire v5 TPU ( https://cloud.google.com/blog/products/ai-machine-learning/i... ) It's unclear how that compares to a high-end consumer GPU like a 3090, but they seem to have similar INT8 TFLOPS. The TPU has less memory (16 vs. 24), and I'm unsure of the other specs. Something doesn't add up, in my opinion, thou…

There's been a lot of work in optimising inference speed of SD - SD Turbo, latent consistency models, Hyper-SD, etc. It is very possible to hit these frame rates now.

Re: Diffusion models are real-time game engines

#403

Post by godelski »

_hark wrote: post #391
godelski wrote: post #332

Earlier quoted context omitted.

I think your confusing capacity with the training dynamics. Capacity is autological. The amount of information it can express. Training dynamics are the way the model learns, the optimization process, etc. So this is where things like regularization come into play. There's also architecture which affects the training dynamics as well as model capacity. Which makes no guarantee that you get the most information dense…

Sorry I wasn't more clear! I'm referring to the Kolmogorov complexity of the network. The OP said: > With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that. And they're not wrong! An ideally trained network could, in principle, learn…

  > An ideally trained network could, in principle, learn the data-generating program
No disagreement

  > I might have a NN that naively looks like it takes up GBs of space, but it might actually be parameterizing a much simpler function (hence our ability to prune/compress the weights without performance loss - most of the capacity wasn't being used for any interesting computation).
Also no disagreement.

I suggested that this probably isn't the case here since they tried distillation and saw no effect. While this isn't proof that this particular model can't be compressed more it does suggest that it is non-trivial. This is especially true given the huge difference in size. I mean we're talking about 700x...

Where I think our disagreement is in that I read the OP as saying __this__ network. If we're talking about a theoretical network, well... nothing I said anywhere is in any disagreement with that. I even said in the post I linked to that the difference shows that there's still a long way to go but that this is still cool. Why did I assume OP was talking about __this__ network? Well because we're in a thread talking about a paper and well... yes, we're talking about compression machines so theoretically (well not actually supported by any math theory) this is true for so many things and that is a bit elementary. So makes more sense (imo) that we're talking about this network. And I wanted to make it clear that this network is nowhere near compression. Can further research later result in something that is better than the source code? Who knows? For all the reasons we've both mentioned. We know they are universal approximators (which are not universal mimicers and have limits) but we have no guarantee of global convergence (let alone proof such a thing exists in many problems).

And I'm not sure why you're trying to explain the basic concepts to me. I mentioned I was an ML researcher. I see you're a PhD at Oxford. I'm sure you would be annoyed if I was doing the same to you. We can talk at a different level.

Re: Diffusion models are real-time game engines

#404

Post by lynx23 »

Hehe, this sounds like the backstory of a remake of the Terminator, or "I have no mouth, but I must scream." In the aftermath of AI killing off humanity, researchers look deeply into how this could have ahppened. And after a number of dead ends, they finally realize: it was trained, in its infancy, on Doom!

Re: Diffusion models are real-time game engines

#406

Post by InDubioProRubio »

throwway_278314 wrote: post #328
dewarrn1 wrote: post #221

Earlier quoted context omitted.

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

Work which exaggerates the blindness. The people were told to focus very deeply on a certain aspect of the scene. Maintaining that focus means explicitly blocking things not related to that focus. Also, there is social pressure at the end to have peformed well at the task; evaluating them on a task which is intentionally completely different than the one explicitly given is going to bias people away from reporting go…

Does that work on autistic people? Having no filters or fewer filters, should allow them to be more efficient "on guard duty" looking for unexpected things.

Re: Diffusion models are real-time game engines

#407

Post by rldjbpin »

zzanz wrote: post #3

The quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.

to me the closer analogy here is the "running minecraft inside minecraft" (https://news.ycombinator.com/item?id=32901461)

Re: Diffusion models are real-time game engines

#409

Post by rldjbpin »

this is truly a cool demo, but a very misleading title.

to me it seems like a very bruteforce or greedy way to give the impression to a user that they are "playing" a game. the difference being that you already own the game to make this possible, but cannot let the user use that copy!

using generative AI for game creation is at a nascent stage but there are much more elegant ways to go about the end goal. perhaps in the future with computing so far ahead that we moved beyond the current architecture, this might be worth doing instead of emulation perhaps.

Re: Diffusion models are real-time game engines

#410

Post by Sharlin »

slashdave wrote: post #56
Sharlin wrote: post #45

Earlier quoted context omitted.

This is not "just" video, however. It's interactive in real time. Sure, you can say that playing is simply video with some extra parameters thrown in to encode player input, but still.

It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.

Uff, I guess you’re right. Mea culpa. I misread their diagram to represent inference when it was about training instead. The latter is conditioned on actions, but… how do they generate the actual output frames then? What’s the input? Is it just image-to-image based on the previous frame? The paper doesn’t seem to explain the inference part at all well :(
Post reply on HN