Check out this subreddit for a decent "source of truth": reddit.com/r/localllama
Ask HN: Which LLMs can run locally on most consumer computers
41–50 of 96 posts
Re: Ask HN: Which LLMs can run locally on most consumer computers
#42See llamafile (https://github.com/Mozilla-Ocho/llamafile), a standalone packaging of llama.cpp that runs an LLM locally. It will use the GPU, but falls back on the CPU. CPU-only performance of small, quantized models is still pretty decent, and the page lists estimated memory requirements for currently popular models.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#43There definitely are smaller LLMs that can run on consumer computers, but as for their performance... You would be lucky to get a full sentence. On the other hand, sending and receiving responses as text is probably the fastest and most realistic way to implement these things in games.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#44Is there any validity to the idea of using a higher-level LLM to generate the initial data, and then copying that data to a lower-level LLM for actual use? For example, another comment asked: "If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally?" So what if you used a paid LLM to analyze thes…
Maybe. You'd need to develop such a "more efficient" format. Turning unstructured text into knowledge graphs has gotten attention lately. Though I'm honestly skeptical of how useful those will turn out to be. Often times you just can't break down unstructured data into structured data without loosing a ton of information.
Turning the data into an intermediary, not directly understandable by humans (say very-high density embeddings) format might be a more promising path.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#45Related question: what's the minimum GPU that's roughly equivalent to Microsoft's Copilot+ spec NPU? I imagine that Copilot+ will become the target minimum spec for many local LLM products and that most local LLM vendors will use GPU instead of NPU if a good GPU is available.
The NPU in the Snapdragon SoC used by the Windows Surface laptops was quoted to be ~ 40 trillion ops/s (TOPS).
Nvidia 4070 Ti has roughly the same performance: https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti.c3...
Of course, I'm massively oversimplifying, but it should be in the ballpark.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#46Related question: what's the minimum GPU that's roughly equivalent to Microsoft's Copilot+ spec NPU? I imagine that Copilot+ will become the target minimum spec for many local LLM products and that most local LLM vendors will use GPU instead of NPU if a good GPU is available.
I was looking out for a new laptop but was wondering the same. This NPU thing might be one of Microsoft's bets that pays off, and makes all pre-NPU hardware obsolete quickly. Though of course they have doubled down on various failed projexts before (arm Windows, windows phones, etc)
Re: Ask HN: Which LLMs can run locally on most consumer computers
#47Maybe a dumb question, but I think anyone reading this question would know a good answer for me. If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally? "Best" in this case would be I would want to get the best/smartest answers from my questions about these PDFs. They're all full-text PDFs, stud…
LlamaIndex can make this task possible in a very few (surprisingly few) lines of code: https://docs.llamaindex.ai/en/stable/understanding/putting_i...
You'll likely want to move beyond the first examples so you can choose models & methods. Either way, LI has tons of great documentation and was originally built for this purpose. They also have a commercial Parsing product with very generous free quotas (last I checked)
Re: Ask HN: Which LLMs can run locally on most consumer computers
#48The general rule is that VRAM == parameter count in billions (I'm generalizing gguf finetunes here)
8GB vram cards can run 7B models
16GB vram cards can run 13B models
24GB vram cards can run up to 33B models
Now to your question, what can most computers run? You need to look at the tiny but specialized models. I would think 3B models could be ran reasonably well even on the CPU. Intellij has a absolutely microscopic quite good and I don't notice any delay.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#49The general rule is that VRAM == parameter count in billions (I'm generalizing gguf finetunes here) 8GB vram cards can run 7B models 16GB vram cards can run 13B models 24GB vram cards can run up to 33B models Now to your question, what can most computers run? You need to look at the tiny but specialized models. I would think 3B models could be ran reasonably well even on the CPU. Intellij has a absolutely microscopic…
Perhaps there's a simple explanation but why does 24GB of VRAM offer such a large relative uplift in parameter count? (is memory bandwidth a factor rather than just the total memory amount?)
Re: Ask HN: Which LLMs can run locally on most consumer computers
#50Is there any validity to the idea of using a higher-level LLM to generate the initial data, and then copying that data to a lower-level LLM for actual use? For example, another comment asked: "If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally?" So what if you used a paid LLM to analyze thes…
Yes, this model works in many cases.
For example, ask the (better, costlier) Claude Opus to generate high-quality prompts, which get fed into (worse, cheaper) Claude Sonnet.