Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a low-latency AI companion that plays Skyrim with me

pantel.is

Post reply on HN
Print view View original

61–70 of 86 posts

Re: I built a low-latency AI companion that plays Skyrim with me

#61

Post by jasondigitized »

threethirtytwo wrote: post #59
utopiah wrote: post #54

Earlier quoted context omitted.

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

Yeah makes sense this is what I said about AI and coding. No way it could do anything complex. Now AI does all the coding for me, I couldn’t have been more wrong. You will be wrong too.

Yeah, so the dog is derpy today, but the possibilities in the future are going to be awesome.

Re: I built a low-latency AI companion that plays Skyrim with me

#62

Post by pantelisk »

mike_hearn wrote: post #25

This is the sort of thing that the GPT-Live model would excel at when it launches. I'm not sure you'd need the ALE model at all if they launch Live in the API with acceptable pricing as it solves the latency issues with voice entirely. It's an impressive piece of engineering that hasn't been discussed much yet, I guess because it's only a ChatGPT feature and not accessible to developers. The model can dispatch tools…

This is part of the reason I wanted to announce it even before I feel ready to ship... The frontier models have such crazy velocity (admirable) and moving so fast that I didn't want them to make an announcement and suddenly trivialize all the work I 've put into this, overnight.

The fun technical challenges (that can also act as any sort of weak moat) are being taken away one by one, on an almost weekly cadence now! :)

Re: I built a low-latency AI companion that plays Skyrim with me

#63

Post by debo_ »

pantelisk wrote: post #36
voidUpdate wrote: post #32

Does it have any agency of its own, or is it entirely reliant on you commanding it? > "Maybe it's because Varkos is a dog, and who doesn't like dogs" Me

Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its…

There are players in my D&D groups that act very similar to your bored dog companion.

Re: I built a low-latency AI companion that plays Skyrim with me

#65

Post by jakzurr »

pantelisk wrote: post #60
utopiah wrote: post #54

Earlier quoted context omitted.

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

So, this is what I have been thinking (and part of the reason why I want to "cook" it more before shipping). Since I have been building it for so long, I am not sure how much "overfit" it is on my data, the way I speak and the phrases I 'm using. And of course the opposite too, how much the dog trained me to speak to it a certain way to maximize outcome success. But then I thought, when people play games they are not…

That's absolutely my thought too.

When watching the vids, I remembered talking (mostly text chat) in MMORPG's 20 years ago. Even when we started using Skype, etc., chatting was frequently short, slang, and quick info. Like, "pulling", "OTM", "BRB", "OOM". Or even just "let's go", and so on.

But with some practice sessions, I suspect this system would pretty quickly gravitate to using any players slang speech. Am I wrong?

Re: I built a low-latency AI companion that plays Skyrim with me

#66

Post by skoocda »

pantelisk wrote: post #27
alwaysanoobie wrote: post #22

Earlier quoted context omitted.

What I don't understand is how are you passing the world state to the model? Say for instance when you ask the dog to do an action when you launch an arrow upwards how is the LLM continuously tracking the state of the game to be able to respond?

At that point it's deterministic and doesn't need the LLM. Once a plan has been decomposed and evaluated, it becomes a sequence of actions. Actions can either wait on a trigger or flow from one to another. For the arrow example: wait here -> bind: arrow event -> pick_up $target -> goto player -> drop $target. A plan can bind to multiple steps and actions. eg you can tell the dog, "when I say apple you say banana" thi…

I think what's still a bit unclear is how the action space gets linked to a plan. Did you create your own action prototypes, or are you using ones exposed via an existing Skyrim world state interface layer?

How often are you running the world JSON through the planner? When do you give up on fuzzy matching if an action doesn't fit?

Do you use any vision tools? such as the plugin from https://github.com/MinLL/SkyrimNet-GamePlugin

Super cool project btw, thanks for sharing!

Re: I built a low-latency AI companion that plays Skyrim with me

#67

Post by [deleted] »

SamBam wrote: post #56
utopiah wrote: post #54

Earlier quoted context omitted.

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do? The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.

[deleted]

Re: I built a low-latency AI companion that plays Skyrim with me

#68

Post by RunSet »

seabombs wrote: post #5

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as t…

> I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game.

https://quake.fandom.com/wiki/CUJO

Re: I built a low-latency AI companion that plays Skyrim with me

#69

Post by kdkdjfjejd »

SamBam wrote: post #56
utopiah wrote: post #54

Earlier quoted context omitted.

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do? The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.

Except it is not a dog. It is an AI. It’s a character in the game.

People often love to fus ro dah Skyrim companions off of cliffs because they can be incompetent annoying little dumb things that won’t shut up.

The character taking on canine form won’t suddenly make gamers more likely to put up with it messing about their gameplay. Thats just not how things work.

Re: I built a low-latency AI companion that plays Skyrim with me

#70

Post by kdkdjfjejd »

jasondigitized wrote: post #61
threethirtytwo wrote: post #59

Earlier quoted context omitted.

Yeah makes sense this is what I said about AI and coding. No way it could do anything complex. Now AI does all the coding for me, I couldn’t have been more wrong. You will be wrong too.

Yeah, so the dog is derpy today, but the possibilities in the future are going to be awesome.

Indeed. Can’t wait until they sell me a subscription to a game that I will just look at while an AI plays and has fun for me. Whee! Awesome times ahead indeed.

Or maybe it won’t play the game for me but will do whatever it wants, mess up about my gameplay because it inferred it should do something I did not order it to do.

Either way, awesome! Lovely gaming experience. I remember being a kid and thinking “wouldn’t it be so much more fun it if I didn’t have to actually play the game?”.

Post reply on HN