Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

World Emulation via Neural Network

madebyoll.in

Post reply on HN
Print view View original

41–48 of 48 posts

Re: World Emulation via Neural Network

#41

Post by Legend2440 »

das_keyboard wrote: post #30

> So, if traditional game worlds are paintings, neural worlds are photographs. Information flows from sensor to screen without passing through human hands. I don't get this analogy at all. Instead of a human information flows through a neural network which alters the information. > Every lifelike detail in the final world is only there because my phone recorded it. I might be wrong here but I don't think this is true…

>It might also be there because the network inferred that it is there based on previous data.

There is no previous data. This network is exclusively trained on the data he collected from the scene.

Re: World Emulation via Neural Network

#42

Post by ollin »

Valk3_ wrote: post #25

This might be a vague question, but what kind of intuition or knowledge do you need to work with these kind of things, say if you want to make your own model? Is it just having experience with image generation and trying to incorporate relevant inputs that you would expect in a 3D world, like the control information you added for instance?

I think https://diamond-wm.github.io is a reasonable place to start (they have public world-model training code, and people have successfully adapted their codebase to other games e.g. https://derewah.dev/projects/ai-mariokart). Most modern world models are essentially image generators with additional inputs (past-frames + controls) added on, so understanding how Diffusion/IADB/Flow Matching work would definitely help.

Re: World Emulation via Neural Network

#43

Post by Valk3_ »

ollin wrote: post #42
Valk3_ wrote: post #25

This might be a vague question, but what kind of intuition or knowledge do you need to work with these kind of things, say if you want to make your own model? Is it just having experience with image generation and trying to incorporate relevant inputs that you would expect in a 3D world, like the control information you added for instance?

I think https://diamond-wm.github.io is a reasonable place to start (they have public world-model training code, and people have successfully adapted their codebase to other games e.g. https://derewah.dev/projects/ai-mariokart ). Most modern world models are essentially image generators with additional inputs (past-frames + controls) added on, so understanding how Diffusion/IADB/Flow Matching work would definitely he…

Thanks!

Re: World Emulation via Neural Network

#44

Post by foxglacier »

stormfather wrote: post #38

Its a time capsule, among other things. I want to take many, many videos of my grandpa's farm, and be able to walk around in it in VR using something like this in the future.

You can do it using the more classic technique of photogrammetry. There are commercial products used by real estate salesmen to produce high quality "games" where you walk around inside a house, but they're more like Google Streetview where you swoosh between points where a 360 degree photo was taken. All those things will be more faithful than neurally generating next frames based on previous frames and control input.

Re: World Emulation via Neural Network

#45

Post by Jotalea »

It's a really interesting project, reminds me of the 360° videos I used to watch on my phone, back in 2015.

But there's one thing that I'm a little bit worried about: I was getting like 8 stable FPS on my 3 years old flagship phone. My concern is that these models are not optimized to run on this type of hardware, which may or may not lead to hardware obsolescence quicker than planned. And it's not like these aren't powerful, they really are.

Re: World Emulation via Neural Network

#46

Post by ollin »

Jotalea wrote: post #45

It's a really interesting project, reminds me of the 360° videos I used to watch on my phone, back in 2015. But there's one thing that I'm a little bit worried about: I was getting like 8 stable FPS on my 3 years old flagship phone. My concern is that these models are not optimized to run on this type of hardware, which may or may not lead to hardware obsolescence quicker than planned. And it's not like these aren't…

Curious, which device/OS/browser? I did all my testing on 4-year old hardware (iPhone 13 Pro, M1 Pro MBP), and the model itself is extremely tiny (~1GFLOP) so I'm optimistic that performance issues would be solvable with a better software stack (e.g. native app).

Re: World Emulation via Neural Network

#47

Post by Jotalea »

ollin wrote: post #46
Jotalea wrote: post #45

It's a really interesting project, reminds me of the 360° videos I used to watch on my phone, back in 2015. But there's one thing that I'm a little bit worried about: I was getting like 8 stable FPS on my 3 years old flagship phone. My concern is that these models are not optimized to run on this type of hardware, which may or may not lead to hardware obsolescence quicker than planned. And it's not like these aren't…

Curious, which device/OS/browser? I did all my testing on 4-year old hardware (iPhone 13 Pro, M1 Pro MBP), and the model itself is extremely tiny (~1GFLOP) so I'm optimistic that performance issues would be solvable with a better software stack (e.g. native app).

I was on my Samsung Galaxy S21FE (Snapdragon 888), on the latest version of the Firefox browser for Android (138.0), on One UI 6.1 (Android 14). It is possibly the most powerful device I own, that's why I was concerned.

Re: World Emulation via Neural Network

#48

Post by ollin »

Jotalea wrote: post #47
ollin wrote: post #46

Earlier quoted context omitted.

Curious, which device/OS/browser? I did all my testing on 4-year old hardware (iPhone 13 Pro, M1 Pro MBP), and the model itself is extremely tiny (~1GFLOP) so I'm optimistic that performance issues would be solvable with a better software stack (e.g. native app).

I was on my Samsung Galaxy S21FE (Snapdragon 888), on the latest version of the Firefox browser for Android (138.0), on One UI 6.1 (Android 14). It is possibly the most powerful device I own, that's why I was concerned.

Got it, that makes sense! In terms of raw compute capability, a Snapdragon 888's GPU should have more than enough power to run this demo smoothly. I think I just need to optimize the inference setup better (maybe switch to WebGPU if the platform supports it?) and do targeted testing on Firefox/Android.
Post reply on HN