Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Which LLMs can run locally on most consumer computers

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 96 posts

Re: Ask HN: Which LLMs can run locally on most consumer computers

#81

Post by IXCoach »

reaperman wrote: post #64
MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

> for games: i think they are simply too slow for now? I think it's two-fold. The primary one is that it's likely very difficult to maintain a designers storyline vision and desired "atmosphere / feel", because LLM's currently "go off the rails" too easily. The second is that the teams with enough funding to properly fine-tune generative AI to do dialog, level/environment-creation, character-generation, etc. that fun…

I agree in the context of LLMs running locally. For API connected games, cloud support for nuanced conversations would be a tremendous value add. Take a hit like Cyberpunk, create a Mod that wires into a custom AI from ixcoach.com... we could literally integrate the most nuanced self inquiry practices into the top games this way.

Anyone working on top games through mods that wants to explore this, let me know, Next AI Labs would be interested in supporting such efforts.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#82

Post by IXCoach »

everforward wrote: post #71
reaperman wrote: post #64

Earlier quoted context omitted.

> for games: i think they are simply too slow for now? I think it's two-fold. The primary one is that it's likely very difficult to maintain a designers storyline vision and desired "atmosphere / feel", because LLM's currently "go off the rails" too easily. The second is that the teams with enough funding to properly fine-tune generative AI to do dialog, level/environment-creation, character-generation, etc. that fun…

I think any NPC with dialogue important to a goal (a quest, a tutorial, etc) is going to be hard to use generative AI for. It not only needs to be coherent with the story, but it needs to correctly include certain ideas. I.e. if the NPC gives a quest to go find some item at some location, it needs to say what the item is and where it is. I think we're currently stuck in a local minima where AI isn't up to the task of…

To some degree, yes. But, theres a low value to cost ratio in that exact UX.

Take a single character in the game, and enable that character the depth and nuance of a true experience between a Zen Master / Inquiry facilitator, powered by AI. IXCoach.com can do a phenomenal job powering this, so literally the only code needed for an MPV is the mod + character api.

Then, the cost benefit ratio is 400x, and in a day of coding you have taken a game that is mostly pure entertainment, and provided a means for depth, nuance and personal development that literally leads the market.

I pinged the executive producer of CD Project Red on this, it's viable.

https://www.linkedin.com/in/danhernberg/

Re: Ask HN: Which LLMs can run locally on most consumer computers

#83

Post by everforward »

reaperman wrote: post #76
everforward wrote: post #71

Earlier quoted context omitted.

I think any NPC with dialogue important to a goal (a quest, a tutorial, etc) is going to be hard to use generative AI for. It not only needs to be coherent with the story, but it needs to correctly include certain ideas. I.e. if the NPC gives a quest to go find some item at some location, it needs to say what the item is and where it is. I think we're currently stuck in a local minima where AI isn't up to the task of…

Agreed for current systems. I’m sure we’ll get models in the future which will facilitate this but for now LLMs don’t really stay on task like a professional human would. And even in AI Dungeon the AI plays so fast and loose that it breaks immersion. Like if I’m doing a space trading roleplay, it doesn’t consider things like making sure the product I’m buying selling meets a specific spec, and often a vendor will sta…

I totally think we'll get there, I just don't think we're there yet.

I really think the next step is a heavily AI-integrated version of D&D where the DM can serve as a "filter" for some of the more unhinged output (where appropriate; an intentionally incoherent goblin with some text-to-speech could be phenomenal).

I think that's about where we're at, and I'm expecting a wave of "AI-enhanced" D&D apps any day now. They probably already exist and I just haven't seen them. I would imagine there are still occasional issues with the AI utterly choking; I see it every once in a while on some of my more "fantasy" prompts where I get too specific and it just ignores what I asked.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#84

Post by FezzikTheGiant »

phi-go wrote: post #77
MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

There are a few games that use LLMs and voice, they are usually hilariously janky.

Could you name some?

Re: Ask HN: Which LLMs can run locally on most consumer computers

#85

Post by imtringued »

MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

The problem is that you need two GPUs and the AI one can't be from AMD. We aren't 15 years away. More like two or three. NPUs are coming and DDR6 plus quad channel memory would get you decent performance on small LLMs like llama3.

You're also forgetting that batch performance is already an order of magnitude better than single session inference.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#87

Post by imtringued »

FezzikTheGiant wrote: post #5
Isuckatcode wrote: post #4

I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air. [1] https://ollama.com

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

There isn't. For games you would need vLLM, because batch size is more important than latency. Something that people don't seem to understand is that an NPC doesn't need to generate tokens faster than its TTS can speak. You only need to minimize the time to first token.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#88

Post by imtringued »

bryanlarsen wrote: post #33

Related question: what's the minimum GPU that's roughly equivalent to Microsoft's Copilot+ spec NPU? I imagine that Copilot+ will become the target minimum spec for many local LLM products and that most local LLM vendors will use GPU instead of NPU if a good GPU is available.

Basically any GPU with at least 32GB RAM and 12 TFLOPs.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#89

Post by imtringued »

sn0wr8ven wrote: post #43

There definitely are smaller LLMs that can run on consumer computers, but as for their performance... You would be lucky to get a full sentence. On the other hand, sending and receiving responses as text is probably the fastest and most realistic way to implement these things in games.

I've gone past the 8k context window with very good text generation on llama3. I don't know what you're smoking.
Post reply on HN