Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best hardware to run small/medium models locally?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 99 posts

Re: Ask HN: What's the best hardware to run small/medium models locally?

#71

Post by addandsubtract »

wenc wrote: post #68
addandsubtract wrote: post #67

Earlier quoted context omitted.

I'm getting "Error: llama runner process has terminated" when trying to run the model. According to this ticket[1], it's a memory issue. Not sure why 16GB ram are struggling with a 7b model, though. [1] https://github.com/jmorganca/ollama/issues/1231

Could you try another model like Mistral 7b and see if triggers the same error? (And check your available memory) I have 16GB too and have no trouble running even 13B models.

Oh, interesting. Mistral:7b works, but wizardcoder:7b-python throws the same error as before. What's another good coding model to use besides wizardcoder?

Edit: wizardcoder:7b-python-q4_1 throws the same error

Re: Ask HN: What's the best hardware to run small/medium models locally?

#72

Post by sparker72678 »

steve_adams_86 wrote: post #5

Somewhat related. I’ve got an M2 Max Mac Studio with 32GB of ram. Is there anything interesting I can do with it in terms of ML? What’s the scene like on moderately powered equipment like this?

Try https://ollama.ai

Super easy to get setup and play around. Probably start with Llama2 13b or 7b, and then Mistral 7b.

If you want a dead-simple chat GUI for interfacing, check out Ollamac.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#73

Post by KolenCh »

(If you want a Mac,) Apple silicon has the advantage of the unify memory, and with llama.cpp, they can run those models locally and quickly. I’d say start with the largest model you want to run, run it through llama.cpp which will tell you the amount of memory needed. And buy the Mac with at least that amount of memory that you can afford. If you have more budget, prioritize more memory because you may want to be able to run larger model later.

If not Mac, follow other advice with NVidia GPU. in term of the software ecosystem, NVidia >> Apple >> AMD > Intel. (I think I got the ordering right, but the magnitude of difference might be subjective.)

Re: Ask HN: What's the best hardware to run small/medium models locally?

#76

Post by qingcharles »

l72 wrote: post #75

To add to this, I have a laptop with 32G of RAM and am able to run some 7B models on CPU. But I'd like to work on some larger models. Are there any eGPUs that can aid in this?

I have a 12-year-old desktop with 24GB of RAM and no modern GPU and can run the 7B models, it's just no fun :(

Re: Ask HN: What's the best hardware to run small/medium models locally?

#77

Post by qingcharles »

zvr wrote: post #74

If you're wiling to wait a few days, remember that Intel Core Ultra processors (Meteor Lake) are supposed to be available on December 14th. The embedded NPU should make a difference.

How will the onboard NPU compare to an outboard GPU on AI tasks do you think?

Re: Ask HN: What's the best hardware to run small/medium models locally?

#78

Post by theogravity »

oaththrowaway wrote: post #19

4060Ti w/ 16GB VRAM or 3090 w/ 24 GB VRAM Of course with those you'll also have to spend some money on motherboard, ram, SSD, PSU, CPU, ect. I think the best bang for the buck is probably a Mac studio with as much ram as you can afford. I bought an RTX A2000 (12GB VRAM), and it's fine for 7B models and some 13B models with 4 bit quantization, but I kind of regret not getting something with more VRAM.

Just FYI you can get a hacked 2080 Ti with 22GB for half the cost of a used 3090.

https://news.ycombinator.com/item?id=38573884

Re: Ask HN: What's the best hardware to run small/medium models locally?

#79

Post by evnc »

RandomWorker wrote: post #7

I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD. If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.

Have you looked into quantization? At 8-bit quantization, a 7B model requires ~7GB of RAM (plus a bit of overhead); at 4-bit, it would require around 3.5GB and fit entirely into the RAM you have. Quality of generation does degrade a bit the smaller you quantize, but not as much as you may think.
Post reply on HN