Live data from Hacker News

I built a low-latency AI companion that plays Skyrim with me

pantel.is

31–40 of 86 posts

Re: I built a low-latency AI companion that plays Skyrim with me

#31
Very promising project! The latency is great and looks pretty polished for solo work.

> ALE is designed to be largely invariant to phrasing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it doesn't matter, it will still understand you > It creates embeddings from the full text as well as its extracted structure

I get how you utilize embeddings, but most interesting part for me is how you decompose multiple commands? You decompose sentences before embedding?

Re: I built a low-latency AI companion that plays Skyrim with me

#33
post #16

Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.

LLMs have only been really good for a couple years. You can bet that the remaining AAA studios are working hard on figuring this out in current unreleased products.

One challenge I see is that to work well and stand above 'easy' implementations the game would need to provide a rich context for the AI to exist in and interact with. Skyrim is a richly detailed world with an engine that allows that level of interaction, especially as it builds upon earlier elder scrolls games. Then there's the factor of whether commercial studios with the capacity to make that world in the first place (which has been a major draw for players) will invest further budget into AI to add more value, and if they have options to see a return on that investment via the main purchase price or additional costs.

Re: I built a low-latency AI companion that plays Skyrim with me

#36

Does it have any agency of its own, or is it entirely reliant on you commanding it? > "Maybe it's because Varkos is a dog, and who doesn't like dogs" Me

Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its default "homeostasis value" but also the curve of which it increases). You can think of it as letting an llm design a behavior graph, and then playing that behavior graph until it changes again.

Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website, at the very end, it chooses to chase down an elk on its own much to my surprise.

After play testing it I had to tone it down quite a bit as it caused continuity issues...

In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by having the llm author something along the lines of "It's nice to be finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.

Re: I built a low-latency AI companion that plays Skyrim with me

#37
post #16

Earlier quoted context omitted.

LLMs have only been really good for a couple years. You can bet that the remaining AAA studios are working hard on figuring this out in current unreleased products.

One challenge I see is that to work well and stand above 'easy' implementations the game would need to provide a rich context for the AI to exist in and interact with. Skyrim is a richly detailed world with an engine that allows that level of interaction, especially as it builds upon earlier elder scrolls games. Then there's the factor of whether commercial studios with the capacity to make that world in the first pl…

just offer a separate subscription to host the companion model. I feel dirty typing this.

Re: I built a low-latency AI companion that plays Skyrim with me

#38

>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it) I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old. unfortun…

As soon as the AI progress 'slows down' a little bit and its a little bit more clear what makes sense of converting into a hardware model, there will be for sure something.

It either might be some neural engine like what Apple is doing and tensor units like what high end android phones already have or something dedicated.

Based on the current capacity issues around the globe, perhaps 1-5 years?

But we have for sure crossed a price point were you just might buy 10 ai credits and will be able to just play 100h without running ot of your ai credits. basic conversations etc. finetuned for a game, doesn't need a frontier model.

Re: I built a low-latency AI companion that plays Skyrim with me

#39
Nice demo, I'm always impressed by making these things low latency enough to be actually interactive. Were you inspired at all by the autonomous vtuber Neuro-sama recently getting some in-game integrations to allow her to play Skyrim on her own? (https://www.youtube.com/watch?v=3o7ORYqLrRw shows the highlights from the first stream.)

Re: I built a low-latency AI companion that plays Skyrim with me

#40

>The game runs on Windows, the audio processing and brain runs on my M4 MacBook. It could all run on Windows (provided there is dedicated ~12gb or more gpu ram for it) I do wonder if this is an avenue for console gaming that might be practical in a few years; AI-centric hardware that might be too beefy or expensive for regular users, but can extend new or existing games. Kinda like the expansion paks of old. unfortun…

> I do wonder if this is an avenue for console gaming that might be practical in a few years;

If the AI Bubble bursts a little and the inflated prices for tech dwindle down to normal maybe, otherwise, it will be too expensive.

Or some other advancement that challenges the giants financially but incentivizes companies to build for local AI usage.

Post reply on HN