I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get about 6 tokens per second with a context window of 8k. It's definitely usable if you're patient. I'm glad I also have a Hugging Face account though.
Ask HN: Which LLMs can run locally on most consumer computers
21–30 of 96 posts
Re: Ask HN: Which LLMs can run locally on most consumer computers
#22Gemma 2B and Phi-3 3B, if you run them at Q4 quantization. I wouldn't bother with anything larger than 4B parameters; you're just not going to be able to reliably expect an end-user to run that size of model on a phone yet.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#23“Caniuse” equivalent for LLMs depending on machine specs would be extremely useful!
Re: Ask HN: Which LLMs can run locally on most consumer computers
#24Earlier quoted context omitted.
Depends of what you want to do!? Just for testing most of the 7B model are a good compromise between quality and performance (speak execution time)
Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right? What I think is, from my limited understanding about this field, if smaller models can run on consumer hardware reliably and speedily that would be a game changer.
> This would virtually make inference free right?
Not really. Inference is never "free" unless you cache the result (which is just a static output) or unless you reduce complexity (which yields procedurally less-usable outputs).
Re: Ask HN: Which LLMs can run locally on most consumer computers
#25Quantized 4/5-bit 8b models with medium-short context might be shippable. Still, it’s going to require a nice GPU for all that RAM. Plus you would have to support AMD—I would experiment with llama.cpp as it runs on many architectures.
Hope your game doesn’t have a big texture budget.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#26I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get about 6 tokens per second with a context window of 8k. It's definitely usable if you're patient. I'm glad I also have a Hugging Face account though.
seconded - IMHO Jan has the cleanest UI and most straightforward setup out of all LLM frontends available now.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#27I imagine you would have to solve some tricky scheduling issues to run an LLM on the GPU while it's also busy rendering the game. Frames need to be rendered at a more or less consistent rate no matter what, but the LLM would likely have erratic, spiky GPU utilisation depending on what the agents are doing, so you would have to throttle the LLM execution very carefully. Probably doable but I don't think there's any existing framework support for that.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#28I imagine you would have to solve some tricky scheduling issues to run an LLM on the GPU while it's also busy rendering the game. Frames need to be rendered at a more or less consistent rate no matter what, but the LLM would likely have erratic, spiky GPU utilisation depending on what the agents are doing, so you would have to throttle the LLM execution very carefully. Probably doable but I don't think there's any ex…
or have 2 gpus
Re: Ask HN: Which LLMs can run locally on most consumer computers
#29Quantized 6-8b models run well on consumer GPUs. My concern would be vram limits given you'll likely be expecting the card to do compute _and_ graphics.
Without a GPU I think it will likely be a poor experience, but it won't be long until you'll have to go out of your way to buy consumer hardware that doesn't integrate some kind of TPU.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#30I imagine you would have to solve some tricky scheduling issues to run an LLM on the GPU while it's also busy rendering the game. Frames need to be rendered at a more or less consistent rate no matter what, but the LLM would likely have erratic, spiky GPU utilisation depending on what the agents are doing, so you would have to throttle the LLM execution very carefully. Probably doable but I don't think there's any ex…
or have 2 gpus
That also works but approximately zero gamers have two discrete GPUs. You can't even rely on users to have an integrated GPU and a discrete GPU, there's a lot of systems which only have one or the other.