Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Which LLMs can run locally on most consumer computers

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 96 posts

Re: Ask HN: Which LLMs can run locally on most consumer computers

#51

Post by wing-_-nuts »

MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

There are mods for skyrim right now that run an NPC's dialog and lore through a small 7B model outputs text dialog. Heck if you wanted you could run a 2B whisper model and get reasonably decent voice output.

It's all very exciting, if a little janky.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#52

Post by resource_waste »

winwang wrote: post #41

Check out this subreddit for a decent "source of truth": reddit.com/r/localllama

Nah, too many fanboys thinking their CPU testing is actually using LLMs.

They will say things like "Its a GPU inside a CPU". No that is the marketers telling you about integrated GPUs.

There is a huge divide between CPU and GPU people. GPU people are doing application. CPU people are... happy that they got anything to run.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#53

Post by wing-_-nuts »

noboostforyou wrote: post #49
wing-_-nuts wrote: post #48

The general rule is that VRAM == parameter count in billions (I'm generalizing gguf finetunes here) 8GB vram cards can run 7B models 16GB vram cards can run 13B models 24GB vram cards can run up to 33B models Now to your question, what can most computers run? You need to look at the tiny but specialized models. I would think 3B models could be ran reasonably well even on the CPU. Intellij has a absolutely microscopic…

Perhaps there's a simple explanation but why does 24GB of VRAM offer such a large relative uplift in parameter count? (is memory bandwidth a factor rather than just the total memory amount?)

So, this is a bit misleading. For whatever reason the models tend to be released in certain parameter sizes. 7B models are popular. The next highest is 13B. There are few in between (some 11B). Likewise the jump from 13 is straight to 33B. You can run finetunes of a 33B model that have been cut down a little and fit them in a 24GB card. Likewise those 13B models running on 16GB cards have a lot of head room. You don't need to run as cut down a model, and you can run it with more context (i.e. the amount of your chat it can hold in memory)

I hope that helps, it's not 1:1, and it's a bit confusing

Re: Ask HN: Which LLMs can run locally on most consumer computers

#54

Post by verdverm »

FezzikTheGiant wrote: post #5
Isuckatcode wrote: post #4

I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air. [1] https://ollama.com

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

> Are they able to run at a good speed?

Not on most consumer computers, which likely lack a dedicated GPU. My M2 struggles (only thing that makes it warm) with a 7B model, but token speed is unbearable. I switched to remote APIs for the speed.

If you are targeting gamers with a GPU, the answer may change, but as others have pointed out, there are numerous issues here.

> This would virtually make inference free right?

Yes-ish, if you are only counting your dollars, however it will slow their computer down and have slow response time, which will impact adoption of your game.

If you want to go this route, I'd start with a 2B sized model, and not worry about shipping it nicely. Get some early users to see if this is the way forward.

I suspect that remote LLM calls with sophisticated caching (cross user / convo / pre-gen'd) is something worth exploring as well. IIRC, people suspected gtp3-turbo was caching common queries and avoided the LLM when it could, for the speed

Re: Ask HN: Which LLMs can run locally on most consumer computers

#55

Post by wkat4242 »

noboostforyou wrote: post #49
wing-_-nuts wrote: post #48

The general rule is that VRAM == parameter count in billions (I'm generalizing gguf finetunes here) 8GB vram cards can run 7B models 16GB vram cards can run 13B models 24GB vram cards can run up to 33B models Now to your question, what can most computers run? You need to look at the tiny but specialized models. I would think 3B models could be ran reasonably well even on the CPU. Intellij has a absolutely microscopic…

Perhaps there's a simple explanation but why does 24GB of VRAM offer such a large relative uplift in parameter count? (is memory bandwidth a factor rather than just the total memory amount?)

Probably quantisation.

I own a 4090 and I can only run very heavily quantised 33B models. It's not really worth it.

My LLM server with 16gb gpu mainly runs llama3 with expanded context window which also costs much more memory.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#56

Post by ultrasaurus »

ynniv wrote: post #42

See llamafile ( https://github.com/Mozilla-Ocho/llamafile ), a standalone packaging of llama.cpp that runs an LLM locally. It will use the GPU, but falls back on the CPU. CPU-only performance of small, quantized models is still pretty decent, and the page lists estimated memory requirements for currently popular models.

+100 to this, I don't think many people reading this thread realize how easy they've made it to run a LLM locally. It's a great start if you want to kick multiple tires (be careful to clean up! the gigs add up).

> wget https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF...

> chmod +x TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile

> ./TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile -ngl 999

https://euri.ca/blog/2024-llm-self-hosting-is-easy-now/

Re: Ask HN: Which LLMs can run locally on most consumer computers

#57

Post by FezzikTheGiant »

talldayo wrote: post #24
FezzikTheGiant wrote: post #12

Earlier quoted context omitted.

Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right? What I think is, from my limited understanding about this field, if smaller models can run on consumer hardware reliably and speedily that would be a game changer.

> This would virtually make inference free right? Not really. Inference is never "free" unless you cache the result (which is just a static output) or unless you reduce complexity (which yields procedurally less-usable outputs).

Can you explain further? Why would it not be free if it's running locally

Re: Ask HN: Which LLMs can run locally on most consumer computers

#58

Post by kkielhofner »

keiferski wrote: post #35

Is there any validity to the idea of using a higher-level LLM to generate the initial data, and then copying that data to a lower-level LLM for actual use? For example, another comment asked: "If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally?" So what if you used a paid LLM to analyze thes…

There is actually a specific approach of this concept for generating synthetic data for training datasets called UDAPDR[0].

It or something like it could likely be applied to any form of generation including what you are describing.

[0] - https://github.com/primeqa/primeqa/tree/4ae1b456dbe9f75276fe...

Re: Ask HN: Which LLMs can run locally on most consumer computers

#59

Post by FezzikTheGiant »

sharpshadow wrote: post #19
FezzikTheGiant wrote: post #5

Earlier quoted context omitted.

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

You could also ship a couple of them and let the game/user choose which one to run depending on the hardware.

This is something I was considering as well - thanks

Re: Ask HN: Which LLMs can run locally on most consumer computers

#60

Post by antisthenes »

MyFirstSass wrote: post #32

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now? I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or nee…

How in the world would this be tested? Anything pertaining to game logic needs to be deterministic.

I can't see LLMs in games being used for anything more than some random NPC voice quips. And whose voice would be used? Would voice actors be okay with this?

There are already too many bad games, we certainly don't need thousands more with AI-generated drivel dialogue, although having human writers is not a panacea either way.

Post reply on HN