Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Which LLMs can run locally on most consumer computers

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 96 posts

Re: Ask HN: Which LLMs can run locally on most consumer computers

#31

Post by alexvitkov »

FezzikTheGiant wrote: post #5
Isuckatcode wrote: post #4

I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air. [1] https://ollama.com

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

The biggest roadblock is not running the model on the user's machine, that's barely an issue with 7B models on a gaming PC. The difficulty is in getting the NPC to take interesting actions with a tangible effect on the game world as a result of their conversation with the player.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#32

Post by MyFirstSass »

I've been curious as to when games would implement any kind of these new technologies, but i think they are simply too slow for now?

I think we're at least 10-15 years from being able to run low latency agents that "rag" themselves into the games they are a part of, where there are 100's of them, some of them NPC's other's controlling some game mechanic or checking if the output from other agents is acceptable or needs to be run again.

At the moment a macbook air 16 gb can run Phi-Medium 14gb, which is extremely impressive, but it's 7 tokens per second, way to slow for any kind of gaming, you need to 100x performance and we need 5+ generations before i can see this happening.

Unless there's some other application?

Re: Ask HN: Which LLMs can run locally on most consumer computers

#33

Post by bryanlarsen »

Related question: what's the minimum GPU that's roughly equivalent to Microsoft's Copilot+ spec NPU?

I imagine that Copilot+ will become the target minimum spec for many local LLM products and that most local LLM vendors will use GPU instead of NPU if a good GPU is available.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#34

Post by manishsharan »

blakesterz wrote: post #18

Maybe a dumb question, but I think anyone reading this question would know a good answer for me. If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally? "Best" in this case would be I would want to get the best/smartest answers from my questions about these PDFs. They're all full-text PDFs, stud…

If its just for you, may I suggest Open AI's python notebook examples. This was the one I used to get started.

https://cookbook.openai.com/examples/parse_pdf_docs_for_rag

There are several other examples like this .. but I got stuck in jargon of Langchain or LlamaIndex etc..

Re: Ask HN: Which LLMs can run locally on most consumer computers

#35

Post by keiferski »

Is there any validity to the idea of using a higher-level LLM to generate the initial data, and then copying that data to a lower-level LLM for actual use?

For example, another comment asked:

"If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally?"

So what if you used a paid LLM to analyze these PDFs and create the data, but then moved that data to a weaker LLM in order to run question-answer sessions on it? The idea being that you don't need the better LLM at this point, as you've already extracted the data into a more efficient form.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#36

Post by abdullin »

keiferski wrote: post #35

Is there any validity to the idea of using a higher-level LLM to generate the initial data, and then copying that data to a lower-level LLM for actual use? For example, another comment asked: "If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally?" So what if you used a paid LLM to analyze thes…

Yes, this can work. I’ve done that in a few cases.

In fact, if you split data preprocessing in small enough steps, they could also be run on weaker LLMs. It would take a lot more time, but that is doable.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#37

Post by thibaut_barrere »

keiferski wrote: post #35

Is there any validity to the idea of using a higher-level LLM to generate the initial data, and then copying that data to a lower-level LLM for actual use? For example, another comment asked: "If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally?" So what if you used a paid LLM to analyze thes…

Yes, that is what I am doing on some projects

Re: Ask HN: Which LLMs can run locally on most consumer computers

#39

Post by solardev »

blakesterz wrote: post #18

Maybe a dumb question, but I think anyone reading this question would know a good answer for me. If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally? "Best" in this case would be I would want to get the best/smartest answers from my questions about these PDFs. They're all full-text PDFs, stud…

Not self hosted, but Google Notebook LLM is OK at that: https://notebooklm.google.com/

You can also upload files to ChatGPT and ask questions about it.

Post reply on HN