Live data from Hacker News

I built a low-latency AI companion that plays Skyrim with me

pantel.is

51–60 of 86 posts

Re: I built a low-latency AI companion that plays Skyrim with me

#51
post #3

I like the approach here. It’s cleverer than just “dump context to LLM”. I couldn’t quite figure out if it genuinely runs purely locally though. If nothing else - this is how NPCs should work in games moving forward!

I've been eagerly awaiting the LLM driven NPC revolution in gaming!

Re: I built a low-latency AI companion that plays Skyrim with me

#53
Some variant of this idea has played in the back of my mind over the last few years. I'd love to play Skyrim, but where each character builds their own context for their history, their personality, their goals, and their interactions with you. The world would feel incredibly alive.

Re: I built a low-latency AI companion that plays Skyrim with me

#54
post #5

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as t…

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable.

I do think it is an interesting exploration but the jagged frontier makes it really challenging to know what will consistently work and what will not, to the point that I bet (maybe pessimistically) one will be gradually less daring with creative plan with their "companion" simply because they can't trust it.

Re: I built a low-latency AI companion that plays Skyrim with me

#55

Video game seem like the perfect fit for llm use and I don't understand why it hasn't happened already.

IMHO at least 2 reasons, the jagged frontier (you cant easily evaluate the capabilities of your companion, I expanded a bit on https://news.ycombinator.com/item?id=49419952 ) and probably riskier in a lot of story games the risk of spoilers if the model isn't sandboxed properly, which I don't think is trivial to do as you don't know the training set which probably includes games and books games might be based on.

Re: I built a low-latency AI companion that plays Skyrim with me

#56
post #54
post #5

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as t…

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do?

The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.

Re: I built a low-latency AI companion that plays Skyrim with me

#57
post #56
post #54

Earlier quoted context omitted.

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

I feel like that's the benefit to the creator having made this a dog (even if a demon-possessed dog). As intelligent as a dog can be, can we ever truly understand why they do the things they do? The Uncanny Valley effect would be too big if a human companion randomly started talking to you while you were sneaking up on an enemy. But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog.

"But a dog? You'd be pissed off at it for a bit, and then forgive it because it's a dog."

Some grumpy old school hunters (in less civilized areas) would literally shoot such a dog though for barking in the wrong moment.

Re: I built a low-latency AI companion that plays Skyrim with me

#58
post #54
post #5

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as t…

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

Kinda in the same vein, been reading/listening to Dungeon Crawler Carl lately. One of the main characters is a tortoise shell cat that gains sentience. One of the more fun aspects in the early books is the main character Carl dealing with that cat being loud, not aware of her surroundings, and making it difficult to sneak around an RPG style dungeon.

Also the author really gets the "tortitude" right.

Re: I built a low-latency AI companion that plays Skyrim with me

#59
post #54
post #5

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as t…

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

Yeah makes sense this is what I said about AI and coding. No way it could do anything complex.

Now AI does all the coding for me, I couldn’t have been more wrong. You will be wrong too.

Re: I built a low-latency AI companion that plays Skyrim with me

#60
post #54
post #5

I must say absolutely hilarious video. The "persona" of the dog is great. As someone who is generally pretty "keep your AI out of my art" this looks very fun to play. I can imagine this being cleverly integrated as a primary feature of a game (this must already be in the works). Ideally as a small model able to run locally alongside the game. The moment where the dog is going on about "something foul in the air" as t…

I get it is fun to play for an hour or so then you inexorably encounter behaviors that are too complex to handle, e.g. you try to sneak in on an enemy but get spotted because the dog started talking or barking randomly, then suddenly wonder how much "it" actually gets it. You then start with less and less complex plans... until you realize that direct commands or shortcuts are more reliable. I do think it is an inter…

So, this is what I have been thinking (and part of the reason why I want to "cook" it more before shipping). Since I have been building it for so long, I am not sure how much "overfit" it is on my data, the way I speak and the phrases I 'm using.

And of course the opposite too, how much the dog trained me to speak to it a certain way to maximize outcome success.

But then I thought, when people play games they are not using highly sophisticated vocabulary and there is probably lots of repetition since they are always under some form of multi-tasking stress (playing and replying/speaking). So maybe... maybe, the system can adjust itself. Use a big LLM offline to say "user said X, we did Y - was that good?" - then retrain itself.

The decomposer is basically a bunch of old-school embeddings/classifiers stitched together, it can train super fast and doesn't need tons of data. Could the thing calibrate itself to the user? Does it even need to? (because as I said I 'm a datapoint of 1 and I am not ready for the potentially huge stream of bug reports when I ship (add some perfectionism to the mix and you get the idea)).

edit: typos

Post reply on HN