Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rendering a game in real time with AI

blog.jeffschomay.com

Post reply on HN
Print view View original

81–90 of 115 posts

Re: Rendering a game in real time with AI

#81

Post by thwarted »

BizarroLand wrote: post #74
coolKid721 wrote: post #70

Earlier quoted context omitted.

Compared to just coming up with and having solid systems for generating game assets? Actually having decent quality style consistent 3d models, texture work, animation, sound effects, etc (especially if it were built into say a game engine or something) would actually revolutionize indie game dev. Games are fundamentally artistic works so yes anything decent will actually require tailoring and crafting, AI set up to…

I think you're making a lot of good points for the current SOTA. That being said, it took us a few hundred years all in all just to work out paint, so if people keep working with this tech eventually a game designer could, in theory, lay out the skeleton of a game and tell an AI to do the rest, give it pointers and occasional original art to work into the system, and ship a completed playable game in days. Whether it…

> lay out the skeleton of a game and tell an AI to do the rest, give it pointers and occasional original art to work into the system, and ship a completed playable game in days

"But, think of the indie game designer!" is getting to be quite the take.

We have a machine that produces slop and the selling point is how fast it produces it? And how more people should be using it to spend less time on creative aspects? Would the world be a better place if GRRM "finished" his most well-known work sooner rather than never?

Something about the phrase "tell an AI to do the rest, give it pointers" reminds me of "The Sorcerer's Apprentice" from Fantasia. Not in the surface level dire-warning about laziness, automation, and losing control that story is telling but in that Mickey didn't spend any time thinking about what he was doing and the Wizard's disappointment at the end.

Re: Rendering a game in real time with AI

#82

Post by mediaman »

doawoo wrote: post #72

[flagged]

It's interesting to see software engineers realize that AI can be useful in the hands of competent engineers, but that LLMs tend to produce a mess in the hands of those with little software engineering knowledge. Then they're confused why generative AI asset creation doesn't look that good in the hands of people who have no art training.

I know someone creating a game, and she is using AI in asset creation. But she's also a highly skilled environment technical artist from the industry, and so the use of AI looks totally different than all these demos from non-artists: different types of narrow AI get injected into different little parts of the asset creation pipeline, and the result is a mix of traditional tools (Substance Designer + Painter, Blender, Maya) with AI support in moodboarding, geometry creation and some parts of texture creation. The result is a 2-5x speedup, but instead of looking like slop it looks like a stylistically distinctive, cohesive world with consistent art direction.

The common pattern is that people think AI will automate "other people," because they see its shortcomings in their own field. But because they don't understand the technical skill required in other fields, they assume AI will just "do it." Instead, it seems like AI can be a force multiplier for technically skilled people, but that it begins showing its weakness when asked to take over entire pipelines normally produced by technically skilled people, whether they be engineers or artists.

Re: Rendering a game in real time with AI

#83

Post by lukan »

AnotherGoodName wrote: post #75
pololeono wrote: post #53

Earlier quoted context omitted.

Yes, go to rome. I will be impressed when we have self driving cars in rome.

They are in San Francisco today so it's not like they are doing this on easy mode.

But are they doing it without human intervention?

Re: Rendering a game in real time with AI

#84

Post by sovietmudkipz »

Sci-fi readers who’ve read Ender’s Game will recognize this style of software as similar in concept to the Mind Game Ender Wiggins plays. In the book, the Mind Game renders a game world based on the subject’s mind (conscious, subconscious) in a mechanically similar way to how dreams work for us IRL.

I’m excited for AI rendered games.

Re: Rendering a game in real time with AI

#85

Post by corysama »

sensanaty wrote: post #73
corysama wrote: post #68

Earlier quoted context omitted.

Here’s a recent demo made by researchers at Nvidia trying to render the most detailed, realistic scene possible using AI tech to a small degree -mostly as asset compression, but also in material modeling beyond what direct approaches can currently handle in real time: https://m.youtube.com/watch?v=0_eGq38V1hk Here’s a video from a rando on Reddit conveniently posted today after playing around for an afternoon: https:…

There's something so unsettling about that 2nd link you posted, and I don't mean from an "AI is impressive" POV (I think it looks like absolute garbage but will probably continue improving bit by bit). There's something even in the still frames which scratches my brain in a very unpleasant way, it's hard to describe the sensation. The closest thing is "disgust", maybe it's a uncanny valley type of effect. Also, how a…

> how are these 2 comparable at all. the latter looks like a terrible travel agency ad.

Don't focus on the emotional context of the scenes. Look at the physical content. They are both contain stonework, plants and a human. They are both large, detailed scenes lit by strong sunlight and a bright sky. As far as rendering technology requirements go, they are very similar.

> scratches my brain in a very unpleasant way, it's hard to describe the sensation. The closest thing is "disgust"

There is strong discontinuous motion effect because of how the video tech is based on a sequences of "first frame, last frame" inputs spliced together. There are a couple seconds where her behavior gets uncanny valley --particularly her hands on the door. Wouldn't be a concern in an unrealistic video game. But, would be in a this-realistic game regardless of the tech. And, there is a very slight warble in the fine details. But, you have to really look for those to distinguish them from MPEG artifacts.

I expect what a lot of people here are feeling when they watch the video is disgust based 90% or more just on the fact that the video was AI generated :/

Re: Rendering a game in real time with AI

#86

Post by doawoo »

mediaman wrote: post #82
doawoo wrote: post #72

[flagged]

It's interesting to see software engineers realize that AI can be useful in the hands of competent engineers, but that LLMs tend to produce a mess in the hands of those with little software engineering knowledge. Then they're confused why generative AI asset creation doesn't look that good in the hands of people who have no art training. I know someone creating a game, and she is using AI in asset creation. But she's…

Really interesting perspective honestly, I legitimately have no issue with machine learning being used as a tool, like all technology it has benefits!

It’s when it’s used as a replacement for creativity that really gets to me.

Re: Rendering a game in real time with AI

#87

Post by jschomay »

steveruizok wrote: post #13

We did a similar thing at tldraw with Draw Fast ( https://drawfast.tldraw.com/ ) and it was very fun. Inspired a few knock offs too. We had to shut it down because it was getting popular on Russian Reddit. A related project Lens ( https://lens.tldraw.com ) also used the same technique, but in a collaborative drawing app. At the peak, when we were streaming back video from Fal and getting <100ms of lag, the setup prod…

OP here, I remember both of the draw fast and lens demos! I'm pretty sure those were in the back of my subconscious, inspiring me to explore my take on real time rendering. Thanks for sharing your similar experience. I agree, this was a lot of fun to work on, and like one of the other commenters pointed out, experiencing it viscerally is a whole new kind of feeling, even with the consistency issues. I'd also like to see more experiments on what new kinds of UX could be possible with this tech.

Re: Rendering a game in real time with AI

#88

Post by jschomay »

echelon wrote: post #14
panki27 wrote: post #11

I'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card. While it might take a bit longer to generate, you're still saving network and authentication latency.

This was built in September 2022, and it's still pretty mind-blowing today: https://madebyoll.in/posts/game_emulation_via_dnn/demo/ https://madebyoll.in/posts/game_emulation_via_dnn/ Hook world state up to a server and you have multiplayer. 2025 update: https://madebyoll.in/posts/world_emulation_via_dnn/ https://madebyoll.in/posts/world_emulation_via_dnn/demo/

That's really cool, thanks for sharing!

Re: Rendering a game in real time with AI

#89

Post by jschomay »

sjsdaiuasgdia wrote: post #2

The "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering. The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of…

OP here. Thanks for the feedback. I agree that frame to frame consistency is quite bad currently. I did address that in the post, hinting at some of the techniques others have mentioned here, like in/out-painting and masking previous frames. For me, the exciting parts of this experiment was finding the opportunities and limits of realtime generation, and exploring ways of grounding generated content in a solid yet player controlled world layer.

Re: Rendering a game in real time with AI

#90

Post by elestor »

coolKid721 wrote: post #23

I do not get the point of this at all, why not just generate game assets and run them in an engine? With this format there would be no regularity that the thing you saw before will look the same (and that is not a fixable problem). Actually figuring out and improving AI approaches for generating consistent and decent quality game assets is actually something that will be useful, this I have no idea the point of past…

> I do not get the point of this at all

I think the point of this is just because it's cool. So, as you said, it only serves as a tech demo, but why not? Many things have no point. It's unreasonable, but it's cool.

Post reply on HN