I would like to really just run OpenwebUI and a few models for local chat use. I'm not into training (yet) and am patient- what is a good cost effective way to get started?
What is a good machine to run LLM's on?
1–5 of 5 posts
Re: What is a good machine to run LLM's on?
#2A cheap Nvidia GPU with lots of VRAM, like the 3060 12gb model. About the fastest you can expect for the lowest amount of money.
Re: What is a good machine to run LLM's on?
#3VRAM is king if you want to run larger (and therefore more accurate) models. 12 GB VRAM will let you run 13B models, which are great for local chat, but you could get away with 8 GB VRAM to run an 8B model as well; I'd recommend Llama 3 8B for that.
Re: What is a good machine to run LLM's on?
#4Thank you!
Re: What is a good machine to run LLM's on?
#5Any M-series Mac with RAM larger than the model you want to run on it.