Are there any? I was thinking about LLM based agents and games and this will probably only be viable when most devices can handle LLMs running locally.
Ask HN: Which LLMs can run locally on most consumer computers
1–10 of 96 posts
Re: Ask HN: Which LLMs can run locally on most consumer computers
#2I assume the question is rather which LLM can cover most of the tasks while delivering decent quality. I would prefer an architecture using different LLM for different tasks rather like 'specialists' instead of simple 'agents'. I used to take the main task and divide it in smaller tasks and see what can I use to solve the problem. Sometimes rule-based approaches can be already enough for a sub-task and LLM would be not only overkill but also more difficult to implement and maintain.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#3I assume the question is rather which LLM can cover most of the tasks while delivering decent quality. I would prefer an architecture using different LLM for different tasks rather like 'specialists' instead of simple 'agents'. I used to take the main task and divide it in smaller tasks and see what can I use to solve the problem. Sometimes rule-based approaches can be already enough for a sub-task and LLM would be n…
so what is your answer to the question?
Re: Ask HN: Which LLMs can run locally on most consumer computers
#4I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#5I was able to successfully run Llama 3 8B, mistral 7B, phi and other 7B models using Ollama [1] on my M1 MacBook Air. [1] https://ollama.com
Are they able to run at a good speed? I'm just wondering what the economics would look like if I want to create agents in my games. I don't think many are going to be willing to get with usage based / token based pricing. That's the biggest roadblock with building LLM-based games right now.
Is there a way to reliably package these models with existing games and make them run locally? This would virtually make inference free right?
What I think is, from my limited understanding about this field, if smaller models can run on consumer hardware reliably and speedily that would be a game changer.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#6Mistral is pretty good, and delivers solid results.
Re: Ask HN: Which LLMs can run locally on most consumer computers
#7Mistral is pretty good, and delivers solid results.
Interesting - is it viable do you think to package a llm like that with an existing game and run it locally - I assume it will be intensive to run but wouldn't that eliminate inference costs?
Re: Ask HN: Which LLMs can run locally on most consumer computers
#8Mistral is pretty good, and delivers solid results.
Interesting - is it viable do you think to package a llm like that with an existing game and run it locally - I assume it will be intensive to run but wouldn't that eliminate inference costs?
Why would you get costs with a local model?
Re: Ask HN: Which LLMs can run locally on most consumer computers
#9Earlier quoted context omitted.
Interesting - is it viable do you think to package a llm like that with an existing game and run it locally - I assume it will be intensive to run but wouldn't that eliminate inference costs?
Why would you get costs with a local model?
yeah that's what I'm saying - it would eliminate inference costs. What I was asking is how feasible is it to package these local llms with another standalone app. For ex. a game
Re: Ask HN: Which LLMs can run locally on most consumer computers
#10Earlier quoted context omitted.
Why would you get costs with a local model?
yeah that's what I'm saying - it would eliminate inference costs. What I was asking is how feasible is it to package these local llms with another standalone app. For ex. a game
Oh sorry. Hm..I actually have no idea. It sounds like a neat idea though. :)