Amazing work. Could you elaborate on the model architecture and the process that lead you to using this architecture?
World Emulation via Neural Network
31–40 of 48 posts
Re: World Emulation via Neural Network
#32> So, if traditional game worlds are paintings, neural worlds are photographs. Information flows from sensor to screen without passing through human hands. I don't get this analogy at all. Instead of a human information flows through a neural network which alters the information. > Every lifelike detail in the final world is only there because my phone recorded it. I might be wrong here but I don't think this is true…
>I don't get this analogy at all. Instead of a human information flows through a neural network which alters the information.
These days most photos are also stored using lossy compression which alters the information.
You can think of this as a form of highly lossy compression of an image of this forest in time and space.
Most lossy compression is 'subtractive' in that detail is subtracted from the image in order to compress it, so the kind of alterations are limited. However there have been previous non-subtractive forms of compression (eg, fractal compression) that have been criticised on the basis of making up details, which is certainly something that a neural network will do. However if the network is only trained on this forest data, rather than being also trained on other data and then fine tuned, then in some sense it does only represent this forest rather than giving an 'informed impression' like a human artist would.
Re: World Emulation via Neural Network
#33Earlier quoted context omitted.
Yeah, the similarities to psychedelics with some of this stuff is remarkable.
It makes me think that maybe our visual perception is similar to what this program is doing in some ways. I wonder if there are any computer vision projects that take a similar world emulation approach? Imagine you collected the depth data also.
Yes the model is a U-Net, which is a type of Convolutional Neural Network (CNN), which is inspired by the structure of the visual cortex.
https://en.wikipedia.org/wiki/Convolutional_neural_network#H...
Re: World Emulation via Neural Network
#34Awesome work / demo / blog
Link to the demo in case people miss it [1]
> using a customized camera app which also recorded my phone’s motion
Using phone's gyro as a proxy for "controls" is very clever
[1] https://madebyoll.in/posts/world_emulation_via_dnn/demo/
Re: World Emulation via Neural Network
#35Amazing work. Could you elaborate on the model architecture and the process that lead you to using this architecture?
The model seems to be viewable here:
https://netron.app/?url=https://madebyoll.in/posts/world_emu...
Re: World Emulation via Neural Network
#36This is very impressive for a hobby project. I was wondering if you were planning to release the source code. Being able to create client-hosted, low-requirement neural networks for world generation could be really useful for game dev or artistic projects.
Yes please! I would love to try and use this on disappearing neighbourhoods, the results are so dreamlike, or like memories!
Re: World Emulation via Neural Network
#37I think this is very interesting because you seem to have reinvented NeRF, if I’m understanding it correctly. I only did one pass through but it looks at first glance like a different approach entirely. More interesting is that you made an easy to use environment authoring tool that (I haven’t tried it yet) seems really slick. Both of those are impressive alone but together that’s very exciting.
NeRF is a more complex and constrained approach, based on a kind of ray tracing. But results are obviously similar.
Right which is why i said it’s an entirely different approach but results in almost the same kind of output
Re: World Emulation via Neural Network
#38Its a time capsule, among other things. I want to take many, many videos of my grandpa's farm, and be able to walk around in it in VR using something like this in the future.
Re: World Emulation via Neural Network
#39It would be quite interesting to try to mess with the neural representations do add or remove the images of some objects there. I'm also curious if the topology of the actual place is similar to the topology of the embedding space.
Re: World Emulation via Neural Network
#40> So, if traditional game worlds are paintings, neural worlds are photographs. Information flows from sensor to screen without passing through human hands. I don't get this analogy at all. Instead of a human information flows through a neural network which alters the information. > Every lifelike detail in the final world is only there because my phone recorded it. I might be wrong here but I don't think this is true…
>I don't get this analogy at all. Instead of a human information flows through a neural network which alters the information. These days most photos are also stored using lossy compression which alters the information. You can think of this as a form of highly lossy compression of an image of this forest in time and space. Most lossy compression is 'subtractive' in that detail is subtracted from the image in order to…
>These days most photos are also stored using lossy compression which alters the information.
I noticed this in some photos I see online starting maybe 5-10 years ago.
I'd click through to a high res version of the photo, and instead of sensor noise or jpeg artefacts, I'd see these bizarre snakelike formations, as though the thing had been put through style transfer.