Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Which LLMs can run locally on most consumer computers

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 96 posts

Re: Ask HN: Which LLMs can run locally on most consumer computers

#11

Post by calculito »

rahimrezgui wrote: post #3
calculito wrote: post #2

I assume the question is rather which LLM can cover most of the tasks while delivering decent quality. I would prefer an architecture using different LLM for different tasks rather like 'specialists' instead of simple 'agents'. I used to take the main task and divide it in smaller tasks and see what can I use to solve the problem. Sometimes rule-based approaches can be already enough for a sub-task and LLM would be n…

so what is your answer to the question?

Depends of what you want to do!? Just for testing most of the 7B model are a good compromise between quality and performance (speak execution time)

Re: Ask HN: Which LLMs can run locally on most consumer computers

#12

Post by FezzikTheGiant »

calculito wrote: post #11
rahimrezgui wrote: post #3

Earlier quoted context omitted.

so what is your answer to the question?

Depends of what you want to do!? Just for testing most of the 7B model are a good compromise between quality and performance (speak execution time)

Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right?

What I think is, from my limited understanding about this field, if smaller models can run on consumer hardware reliably and speedily that would be a game changer.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#13

Post by Isuckatcode »

FezzikTheGiant wrote: post #5
Isuckatcode wrote: post #4

I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air. [1] https://ollama.com

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

Here [1] is a reference to the token/sec of Llama 3 on different apple hardware. You can evaluate if this is an acceptable performance for your agents. I would assume the token/sec would be much lower if the LLM agent is running along the side as the game would also be using a portion of the CPU and GPU. I think this is something that you need to test out on your own to determine its usability.

You can also look into lower parameter models (3B for example) to determine if the balance between accuracy and performance fits under your usecase.

>Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right?

I don't have any knowledge on game dev so I can comment on this but yes, packaging it locally would make the inference free.

[1] https://github.com/ggerganov/llama.cpp/discussions/4167

Re: Ask HN: Which LLMs can run locally on most consumer computers

#14

Post by smarm52 »

Isuckatcode wrote: post #13
FezzikTheGiant wrote: post #5

Earlier quoted context omitted.

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

Here [1] is a reference to the token/sec of Llama 3 on different apple hardware. You can evaluate if this is an acceptable performance for your agents. I would assume the token/sec would be much lower if the LLM agent is running along the side as the game would also be using a portion of the CPU and GPU. I think this is something that you need to test out on your own to determine its usability. You can also look into…

Well since OP doesn't seem to want to: Thank you for your response.

I came across this thread while doing some research, and it's been helpful.

(I hate how common Tragedy of the Commons is. =/)

Re: Ask HN: Which LLMs can run locally on most consumer computers

#15

Post by FezzikTheGiant »

smarm52 wrote: post #14
Isuckatcode wrote: post #13

Earlier quoted context omitted.

Here [1] is a reference to the token/sec of Llama 3 on different apple hardware. You can evaluate if this is an acceptable performance for your agents. I would assume the token/sec would be much lower if the LLM agent is running along the side as the game would also be using a portion of the CPU and GPU. I think this is something that you need to test out on your own to determine its usability. You can also look into…

Well since OP doesn't seem to want to: Thank you for your response. I came across this thread while doing some research, and it's been helpful. (I hate how common Tragedy of the Commons is. =/)

What? chill out buddy - there's such a thing as timezones I was just sleeping

Re: Ask HN: Which LLMs can run locally on most consumer computers

#16

Post by FezzikTheGiant »

Isuckatcode wrote: post #13
FezzikTheGiant wrote: post #5

Earlier quoted context omitted.

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

Here [1] is a reference to the token/sec of Llama 3 on different apple hardware. You can evaluate if this is an acceptable performance for your agents. I would assume the token/sec would be much lower if the LLM agent is running along the side as the game would also be using a portion of the CPU and GPU. I think this is something that you need to test out on your own to determine its usability. You can also look into…

Thanks! This is helpful. I was thinking about the phi models - those might be useful for this task. Will look into how those can be run locally as well

Re: Ask HN: Which LLMs can run locally on most consumer computers

#17

Post by Werewolf255 »

FezzikTheGiant wrote: post #7
jaggs wrote: post #6

Mistral is pretty good, and delivers solid results.

Interesting - is it viable do you think to package a llm like that with an existing game and run it locally - I assume it will be intensive to run but wouldn't that eliminate inference costs?

It would be intensive but it's very doable. You could use koboldcpp or something like that with an exposed endpoint just on the local machine and use that. You'll likely run into issues with GPU vendors and ensuring that you've got the right software versions running, but with some checking, it should be viable. Maybe include a fallback in case the system can't produce results in a timely manner.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#18

Post by blakesterz »

Maybe a dumb question, but I think anyone reading this question would know a good answer for me. If I have a big pile of PDFs and wanted to get an LLM to be really good at answering questions about what's in all those PDFs, would it be best for me to try running this locally? "Best" in this case would be I would want to get the best/smartest answers from my questions about these PDFs. They're all full-text PDFs, studies and results on a specific genetic condition that I'd like to understand better by asking something smart questions.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#19

Post by sharpshadow »

FezzikTheGiant wrote: post #5
Isuckatcode wrote: post #4

I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air. [1] https://ollama.com

Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now. Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inferen…

You could also ship a couple of them and let the game/user choose which one to run depending on the hardware.

Re: Ask HN: Which LLMs can run locally on most consumer computers

#20

Post by spmurrayzzz »

Running them at the edge is definitely possible on most hardware, but not ideal by any means. You'll have to set latency and throughput expectations fairly low if you don't have a GPU to utilize. This is why I'd disagree with your statement re: viability — its really going to be most viable if you centralize the inference in a distributed cloud environment off-device.

Thankfully, between llama 3 8b [1] and mistral 7b [2] you have two really capable generic instruction models you can use out of the box that could run locally for many folks. And the base models are straightforward to finetune if you need different capabilities more specific to your game use cases.

CPU/sysmem offloading is an option with gguf-based models but will hinder your latency and throughput significantly.

The quantized versions of the above models do fit easily in many consumer grade gpus (4-5GB for the weights themselves quantized at 4bpw), but it really depends on how much of your vram overhead you want to dedicate to the model weights vs actually running your game.

[1] https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

[2] https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

Post reply on HN