Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rendering a game in real time with AI

blog.jeffschomay.com

Post reply on HN
Print view View original

101–110 of 115 posts

Re: Rendering a game in real time with AI

#101

Post by pjob »

g105b wrote: post #6

I've been trying to achieve the opposite of this project: render scenes in ASCII/ANSI in the style of old BBS terminal games. I've had terrible success so far. All the AI models I've tried only understand the concept of "pixel art" and not ASCII/ANSI graphics such as what can be seen on https://www.bbsing.com/ , https://16colo.rs , or on Reddit's r/ANSIart/ . If anyone has any tips for how I could achieve this, I wou…

I suspect you'll have much better luck directly implementing the style you want rather than trying to get a (current gen) AI to reliably generate ANSI art. This video is a nice overview of implementing a variation of the style in a post-processing shader:

https://www.youtube.com/watch?v=gg40RWiaHRY

Re: Rendering a game in real time with AI

#102

Post by dvrp »

echelon wrote: post #22
steveruizok wrote: post #13

We did a similar thing at tldraw with Draw Fast ( https://drawfast.tldraw.com/ ) and it was very fun. Inspired a few knock offs too. We had to shut it down because it was getting popular on Russian Reddit. A related project Lens ( https://lens.tldraw.com ) also used the same technique, but in a collaborative drawing app. At the peak, when we were streaming back video from Fal and getting <100ms of lag, the setup prod…

LCM is what Krea used to gain massive momentum and raise their first $30M. The tactile reaction to playing with this tech is that it feels utterly sci-fi. It's so freaking cool. Watching videos does not do it justice. Not enough companies or teams are trying this stuff. This is really cool tech, and I doubt we've seen the peak of what real time rendering can do. The rendering artifacts and quality make the utility fo…

hey thanks for the kind words (krea cofounder here). i agree with that! everyone’s focused on leaderboards and evals but not enough on higher order repercussions imo

on a related (but shameless plug) note, did you check our real time video rendering release?

> https://x.com/krea_ai/status/1961074072487620635

Re: Rendering a game in real time with AI

#103

Post by steveruizok »

xienze wrote: post #76
Topfi wrote: post #48

Earlier quoted context omitted.

They've already shared it under a none-commercial use license: https://github.com/tldraw/draw-fast The TLDraw team, from what I have seen, is really open in sharing their experiments under their own license. Not FOSS strictly, but I feel their licensing approach and decision is fair considering how they fund development, know that it's a contentious topic though for some.

According to that link, you're running the frontend locally but all the work is happening on fal.ai. So the interesting part is not open source.

Here’s a resource for more LCM models, which were built over Stable Diffusion. The inference provider is less a design piece than a detail.

https://huggingface.co/latent-consistency

Re: Rendering a game in real time with AI

#105

Post by johnnyanmac »

johnfn wrote: post #15
sjsdaiuasgdia wrote: post #2

The "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering. The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of…

> The "real-time" version looks awful, etc Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.

criticism is how we grow. And it seems we're more and more afraid for some reason to criticize when "AI" is involved.

Re: Rendering a game in real time with AI

#106

Post by johnnyanmac »

corysama wrote: post #59
roxolotl wrote: post #5

Earlier quoted context omitted.

Why does using a language/vision model feel more “real” to you than using equations which directly describe our understandings of physics?

Because our ability to simulate/render a realistic world in real time using direct equations is still very limited. We’re accustomed to these limitations and often feel “graphics are good enough”. But, we’ll always be decades behind “ILM in real time”. The AI route has a good chance of moving us from decades behind ILM to merely “years behind ILM”.

>The AI route has a good chance of moving us from decades behind ILM to merely “years behind ILM”.

Firstly: we have very accurate models. But not at real time speeds. Games only have some 30,16, or even 11 ms to render a frame. The techniques we have are faking the real physical interactions that render farms can take minutes or hours to pump out per frame.

Secondly: Not at this performance rate. taking 100ms to render a frame is unaccatable in the concept of a game. Games are already so pressed for time budget; unless some hyper JIT happens we can't take all that budget querying an LLM.

Re: Rendering a game in real time with AI

#107

Post by johnnyanmac »

curl-up wrote: post #8
roxolotl wrote: post #5

Earlier quoted context omitted.

Why does using a language/vision model feel more “real” to you than using equations which directly describe our understandings of physics?

Not OP, but I have long thought of this type of approach (underlying "hard coded" object tracking + fuzzy AI rendering) to be the next step, so I'll respond. The problem with using equations is that they seem to have plateaued. Hardware requirements for games today keep growing, and yet every character still has that awful "plastic skin", among all the other issues, and for a lot of people (me included) this creates…

>Hardware requirements for games today keep growing, and yet every character still has that awful "plastic skin", among all the other issues

That's because the number of pixels to render onto keep growing. Instead of focusing on physically based animations and reactions, we chose to leap from 480p to 720p overnight, and then to 1080p in a few more years. Now we quadrupled that and want things with more fidelity with 4x the resolution of last generation.

> images created by image models today look fully realistic.

Because they aren't made in real time (I'll give the BOTD for now and say theya are "fully realistic". Even this sample here claims 100ms. Rendering at 6-7 seconds per frame isn't going to work for any consumer product at any point in gaming history.

>Even if realism is not what you're aiming for, I think it's easy to imagine how this might change the game.

not in real time rendering. I am interested to see if this can help with offline stuff, but we're already so strapped for performance withoout needing to query an "oracle" between frames.

as of now, I'm not even that convinced by the Nvidia 5X series of frame interpolation (which would only be doable by hardware manufactureres).

Re: Rendering a game in real time with AI

#108

Post by johnnyanmac »

tliltocatl wrote: post #96
stego-tech wrote: post #19

Earlier quoted context omitted.

…but then you just have a graphics card, built to render graphics, that you could tap instead through traditional tooling that’s already widely known and which produces consistent output via local assets. While the results of the experiment here are interesting from an academic standpoint, it’s the same issue as remote game streaming: the amount of time you have to process input from the player, render visuals and so…

Nah, vibe coding is the wrongest use of technology, this is way to go. Why? Because good rendering isn't the necessarily most physically accurate one. You might actually want non-realistic rendering, and (depending on the specific style) it might be hard to impossible to get the right one on traditional pipeline. E. g. take "cartoonish" look - toon shading is, frankly said, total crap, because the artists rely on exp…

>Render on a normal pipeline (maybe with no lighting model at all), then put it through a style transfer network.

in 16ms? We keep needing higher resolutions to render to, so I doubt we'll ever have the budget to fit this kind of thing in.

Also, if Toon Shading is crap, this is completely unusable, even as a prototype. The present is being stripped away so it's a bit tiring talking about "the future". Let's actually secure it first.

Re: Rendering a game in real time with AI

#109

Post by johnnyanmac »

nh23423fefe wrote: post #95
bob1029 wrote: post #47

The power consumption of modern gaming is getting a bit out of hand. This AI stuff is taking it to the next level. Ray tracing and other forms of real time global illumination are extremely resource intensive approaches to lighting a scene. Every client machine has to figure out how to light everything every single frame. Contrast this with baked global illumination where the cost is incurred exactly once and is amor…

It doesn't seem like moralizing about resources matters for velocity or capability. > You cannot replace your art team by consuming more electricity This isn't true.

>It doesn't seem like moralizing about resources matters for velocity or capability.

Not until the Hype bubble pops. Then suddenly it'll be all that matters when you no longer have nigh limitless resources.

>This isn't true.

This project is interesting, but it is not making a great arguemnt for this.

Post reply on HN