I'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card.
While it might take a bit longer to generate, you're still saving network and authentication latency.
Rendering a game in real time with AI
11–20 of 115 posts
Re: Rendering a game in real time with AI
#12The "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering. The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of…
It seems it's because OP is generating the whole screen every frame / every move. Of course that will give inconsistent results.
I wonder if this approach would work better:
1. generate the whole screen once
2. on update, create a mask for all changed elements of the underlying data
3. do an inpainting pass with this mask, with regional prompting to specify which parts have changed how
4. when moving the camera, do outpainting
This might not be possible with cloud based solutions, but I can see it being possible locally.
Re: Rendering a game in real time with AI
#13We did a similar thing at tldraw with Draw Fast (https://drawfast.tldraw.com/) and it was very fun. Inspired a few knock offs too. We had to shut it down because it was getting popular on Russian Reddit. A related project Lens (https://lens.tldraw.com) also used the same technique, but in a collaborative drawing app.
At the peak, when we were streaming back video from Fal and getting <100ms of lag, the setup produced one of the most original creative experiences I’d ever had. I wish these sorts of ultra-fast image generators received more attention and research because they do open up some crazy UX.
Re: Rendering a game in real time with AI
#14I'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card. While it might take a bit longer to generate, you're still saving network and authentication latency.
This was built in September 2022, and it's still pretty mind-blowing today:
https://madebyoll.in/posts/game_emulation_via_dnn/demo/
https://madebyoll.in/posts/game_emulation_via_dnn/
Hook world state up to a server and you have multiplayer.
2025 update:
Re: Rendering a game in real time with AI
#15The "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering. The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of…
> The "real-time" version looks awful, etc
Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.
Re: Rendering a game in real time with AI
#16Earlier quoted context omitted.
Why does using a language/vision model feel more “real” to you than using equations which directly describe our understandings of physics?
Not OP, but I have long thought of this type of approach (underlying "hard coded" object tracking + fuzzy AI rendering) to be the next step, so I'll respond. The problem with using equations is that they seem to have plateaued. Hardware requirements for games today keep growing, and yet every character still has that awful "plastic skin", among all the other issues, and for a lot of people (me included) this creates…
You're comparing apples to oranges, holding up today's practical real-time rendering techniques against a hypothetical future neural method that runs many orders of magnitude faster than anything available today, and solves the issues of temporal stability, directability and overall robustness. If we grant "equation based" methods the same liberty then we should be looking ahead to real-time pathtracing research, which is much closer to anything practical than these pure ML experiments.
That's not to say ML doesn't have a place in the pipeline - pathtracers can pair very well with ML-driven heuristics for things like denoising, but in that case the underlying signal is still grounded in physics and the ML part is just papering over the gaps.
Re: Rendering a game in real time with AI
#17I've been trying to achieve the opposite of this project: render scenes in ASCII/ANSI in the style of old BBS terminal games. I've had terrible success so far. All the AI models I've tried only understand the concept of "pixel art" and not ASCII/ANSI graphics such as what can be seen on https://www.bbsing.com/ , https://16colo.rs , or on Reddit's r/ANSIart/ . If anyone has any tips for how I could achieve this, I wou…
Do you mean you want to use AI to generate new scenes in ANSI-art style, or do you mean you want to use AI to render pre-existing scenes as ANSI art?
[deleted]
Re: Rendering a game in real time with AI
#18[dead]
Re: Rendering a game in real time with AI
#19I'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card. While it might take a bit longer to generate, you're still saving network and authentication latency.
…but then you just have a graphics card, built to render graphics, that you could tap instead through traditional tooling that’s already widely known and which produces consistent output via local assets.
While the results of the experiment here are interesting from an academic standpoint, it’s the same issue as remote game streaming: the amount of time you have to process input from the player, render visuals and sound, and transmit it back to the player precludes remote rendering for all but the most latency-insensitive games and framerates. It’s publishers and IP owners trying to solve the problem of ownership (in that they don’t want anyone to own anything, ever) rather than tackling any actually important issues (such as inefficient rendering pipelines, improving asset compression and delivery methods, improving the sandboxing of game code, etc).
Trying to make AI render real-time visuals is the wrongest use of the technology.
Re: Rendering a game in real time with AI
#20This is a pre-cursor to a dystopian future where reality will be a game generated in realtime at 60 FPS and streamed to your brain over Neuralink.