Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Which LLMs can run locally on most consumer computers

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Ask HN: Which LLMs can run locally on most consumer computers

#71

Post by everforward »

reaperman wrote: post #64
MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

> for games: i think they are simply too slow for now? I think it's two-fold. The primary one is that it's likely very difficult to maintain a designers storyline vision and desired "atmosphere / feel", because LLM's currently "go off the rails" too easily. The second is that the teams with enough funding to properly fine-tune generative AI to do dialog, level/environment-creation, character-generation, etc. that fun…

I think any NPC with dialogue important to a goal (a quest, a tutorial, etc) is going to be hard to use generative AI for. It not only needs to be coherent with the story, but it needs to correctly include certain ideas. I.e. if the NPC gives a quest to go find some item at some location, it needs to say what the item is and where it is.

I think we're currently stuck in a local minima where AI isn't up to the task of making a coherent player-interactable world, but an incoherent or fragmented and non-interactable world isn't impressive enough (like No Man's Sky).

Re: Ask HN: Which LLMs can run locally on most consumer computers

#72

Post by wkat4242 »

wing-_-nuts wrote: post #61
wkat4242 wrote: post #55

Earlier quoted context omitted.

Probably quantisation. I own a 4090 and I can only run very heavily quantised 33B models. It's not really worth it. My LLM server with 16gb gpu mainly runs llama3 with expanded context window which also costs much more memory.

Yeah, i have a 3090 and 64gb of ram. I can run a 8x7B and get pretty decent performance out of it with partial offloading.

Really?? For me it's terrible doing that. I also have 64GB RAM but meh. It's so bad when I can no longer offload everything. The tokens literally drizzle in. With full offloading they appear faster than I can read (8B llama3 with 8 bit quant). On a Radeon Pro VII with 16GB (HBM2 memory!)

Re: Ask HN: Which LLMs can run locally on most consumer computers

#73

Post by wing-_-nuts »

wkat4242 wrote: post #72
wing-_-nuts wrote: post #61

Earlier quoted context omitted.

Yeah, i have a 3090 and 64gb of ram. I can run a 8x7B and get pretty decent performance out of it with partial offloading.

Really?? For me it's terrible doing that. I also have 64GB RAM but meh. It's so bad when I can no longer offload everything. The tokens literally drizzle in. With full offloading they appear faster than I can read (8B llama3 with 8 bit quant). On a Radeon Pro VII with 16GB (HBM2 memory!)

Oh man, I hate to say it, but it's likely your amd card. Yes, they can run LLMs and SD, but badly. Larger models are usable for me with partial offloading, but you're right that full loading the model in vram is really preferable.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#74

Post by wkat4242 »

wing-_-nuts wrote: post #73
wkat4242 wrote: post #72

Earlier quoted context omitted.

Really?? For me it's terrible doing that. I also have 64GB RAM but meh. It's so bad when I can no longer offload everything. The tokens literally drizzle in. With full offloading they appear faster than I can read (8B llama3 with 8 bit quant). On a Radeon Pro VII with 16GB (HBM2 memory!)

Oh man, I hate to say it, but it's likely your amd card. Yes, they can run LLMs and SD, but badly. Larger models are usable for me with partial offloading, but you're right that full loading the model in vram is really preferable.

I don't think so, because when I run it on the 4090 I get the same issue (in a system with 5800X3D and 64GB RAM also). I just don't use the 4090 for LLM because I have it for playing VR games and I don't want to tie it up for a 24/7 LLM server :) Also, it's very power-hungry. I do run that one on Windows and the Radeon server is Linux but I don't think that matters a lot. Using the same software stack too (ollama).

In fact the Radeon which cost me only 300 bucks new performs almost as well running LLMs as the 4090 which really surprised me! I think the fast memory (the Radeon has the same 1TB/s memory bandwidth as the 4090!) helps a lot there.

When I run a local model (significantly) bigger than the 24GB VRAM on the 4090 it won't even load for 15 minutes while the 4090 is pegged at 100% all the time. Eventually I just gave up.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#75

Post by wing-_-nuts »

wkat4242 wrote: post #74
wing-_-nuts wrote: post #73

Earlier quoted context omitted.

Oh man, I hate to say it, but it's likely your amd card. Yes, they can run LLMs and SD, but badly. Larger models are usable for me with partial offloading, but you're right that full loading the model in vram is really preferable.

I don't think so, because when I run it on the 4090 I get the same issue (in a system with 5800X3D and 64GB RAM also). I just don't use the 4090 for LLM because I have it for playing VR games and I don't want to tie it up for a 24/7 LLM server :) Also, it's very power-hungry. I do run that one on Windows and the Radeon server is Linux but I don't think that matters a lot. Using the same software stack too (ollama). I…

>When I run a local model (significantly) bigger than the 24GB VRAM on the 4090 it won't even load for 15 minutes while the 4090 is pegged at 100% all the time. Eventually I just gave up.

Yeah the key here is partial offloading. If you're trying to offload more layers than your GPU has memory for, you're gonna have a bad time. I find it kind of infuriating that this is still kind of a black art. There's definitely room for better tooling here.

Regardless, with 24GB of vram, I try to limit my offloading to 20GB and let the rest go to ram. Maybe it's the nature of the 8x7B model I run that makes it better at offloading than other large models. I'm not sure. I wouldn't try the 70B models for sure.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#76

Post by reaperman »

everforward wrote: post #71
reaperman wrote: post #64

Earlier quoted context omitted.

> for games: i think they are simply too slow for now? I think it's two-fold. The primary one is that it's likely very difficult to maintain a designers storyline vision and desired "atmosphere / feel", because LLM's currently "go off the rails" too easily. The second is that the teams with enough funding to properly fine-tune generative AI to do dialog, level/environment-creation, character-generation, etc. that fun…

I think any NPC with dialogue important to a goal (a quest, a tutorial, etc) is going to be hard to use generative AI for. It not only needs to be coherent with the story, but it needs to correctly include certain ideas. I.e. if the NPC gives a quest to go find some item at some location, it needs to say what the item is and where it is. I think we're currently stuck in a local minima where AI isn't up to the task of…

Agreed for current systems. I’m sure we’ll get models in the future which will facilitate this but for now LLMs don’t really stay on task like a professional human would.

And even in AI Dungeon the AI plays so fast and loose that it breaks immersion. Like if I’m doing a space trading roleplay, it doesn’t consider things like making sure the product I’m buying selling meets a specific spec, and often a vendor will start offering to buy Product X from me while I’m negotiating purchasing Product X from them. This "type" of continuity problem happens constantly in AI dungeon.

We’re just not there yet, but I have confidence we’ll get there. I think it’s possible even with our current model/training paradigms but we aren’t using RLHF for game applications yet.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#77

Post by phi-go »

MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

There are a few games that use LLMs and voice, they are usually hilariously janky.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#78

Post by andy_ppp »

abdullin wrote: post #40
andy_ppp wrote: post #23

“Caniuse” equivalent for LLMs depending on machine specs would be extremely useful!

There are too many variables at play, unfortunately. One can ran local LLMs even on RaspberryPi, although it will be horribly slow.

Maybe it wouldn’t be an algorithm, maybe it would be a reporting site where you can review your experience if there’s no way to calculate it.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#79

Post by abdullin »

andy_ppp wrote: post #78
abdullin wrote: post #40

Earlier quoted context omitted.

There are too many variables at play, unfortunately. One can ran local LLMs even on RaspberryPi, although it will be horribly slow.

Maybe it wouldn’t be an algorithm, maybe it would be a reporting site where you can review your experience if there’s no way to calculate it.

LocalLLaMA subreddit usually has some interesting benchmarks and reports.

Here is one example, testing performance of different GPUs and Macs with various flavours of Llama:

https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

Re: Ask HN: Which LLMs can run locally on most consumer computers

#80

Post by artemisart »

kevinkeller wrote: post #45
bryanlarsen wrote: post #33

Related question: what's the minimum GPU that's roughly equivalent to Microsoft's Copilot+ spec NPU? I imagine that Copilot+ will become the target minimum spec for many local LLM products and that most local LLM vendors will use GPU instead of NPU if a good GPU is available.

The NPU in the Snapdragon SoC used by the Windows Surface laptops was quoted to be ~ 40 trillion ops/s (TOPS). Nvidia 4070 Ti has roughly the same performance: https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti.c3... Of course, I'm massively oversimplifying, but it should be in the ballpark.

No, the Nvidia 4070 Ti has much higher performance, TOPS is for integer operations, the 4070 Ti has ~40 float32 TFLOPS and 641 TOPS https://www.nvidia.com/fr-fr/geforce/graphics-cards/40-serie... (which I would say would be peak TOPS for int4 operations, comparing it to the 4080 datasheet, and a bit more than half that for int8 operations) https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid... page 34. I did not find the datasheet for 4070 Ti.
Post reply on HN