Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best hardware to run small/medium models locally?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: Ask HN: What's the best hardware to run small/medium models locally?

#61

Post by wenc »

m1sta_ wrote: post #42
RandomWorker wrote: post #7

I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD. If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.

I run mistral on an M2 air and it's broadly similar to chatgpt.

How? I have an M2 Pro and I run 7B and 13B models through Ollama and also LM Studio.

Because there’s no CUDA, the speed is much slower than ChatGPT. The answers from 7B are also not at the same quality as ChatGPT. (Lots of mistakes and hallucinations)

Re: Ask HN: What's the best hardware to run small/medium models locally?

#62

Post by wenc »

addandsubtract wrote: post #54
adastra22 wrote: post #4

MacBook, thanks to Apple's new MLX framework.

Which model can I use with 16GB ram? I tried Ollama with WizardCoder 7b, but it didn't work.

What didn’t work? I run Ollama on my M2 Pro and all models work.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#64

Post by phanimahesh »

brucethemoose2 wrote: post #53
johndough wrote: post #39

Earlier quoted context omitted.

> Yeah, the 3090 is a meme in local AI communities. Calling the 3090 a "meme" makes it sound like the 3090 is a joke. Do you mean that the 3090 is "well-known" in local AI communities?

Yeah, I just meant that is like the only option, which is crazy because its a 2020 GPU. There are lots of questions about what hardware to get for ML, and the generic answer is basically always "get a 3090." Its so frequently recommended that it feels like a meme to me.

It hits a good sweet spot between price and power, and there's usually some easily available. Availability is a major factor imo.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#65

Post by MPSimmons »

anonzzzies wrote: post #15

Somewhat related; how to run an uncensored model locally? I run llamafile (llamafile-server-0.1-llava-v1.5-7b-q4 and mistral-7b-instruct-v0.1-Q4_K_M-server) ones on my macbook m1 and they run file (fast enough for playing), but they both seem neutered quite a bit. It's hard to get them off the rails and mistral (the above one) actually barfs really quickly just repeating the same letter (fffffff usually) where it sho…

When you're running inference, it's super important to make sure that you're using the right prompt format (if you're using the Oobabooga web text ui, make sure you have 'chat', 'chat-instruct', or 'instruct' properly selected). The model card on Huggingface will usually tell you.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#66

Post by MPSimmons »

oaththrowaway wrote: post #19

4060Ti w/ 16GB VRAM or 3090 w/ 24 GB VRAM Of course with those you'll also have to spend some money on motherboard, ram, SSD, PSU, CPU, ect. I think the best bang for the buck is probably a Mac studio with as much ram as you can afford. I bought an RTX A2000 (12GB VRAM), and it's fine for 7B models and some 13B models with 4 bit quantization, but I kind of regret not getting something with more VRAM.

I hate how the market is right now. I understand that NVidia doesn't want to provide a consumer level graphics card with truly impressive RAM specs, even though they could, because they feel it would eat into their datacenter market (and truth be told, it probably would), but it's super frustrating that you need to pay so much for decent performance, even as a single machine for personal use.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#67

Post by addandsubtract »

wenc wrote: post #62
addandsubtract wrote: post #54

Earlier quoted context omitted.

Which model can I use with 16GB ram? I tried Ollama with WizardCoder 7b, but it didn't work.

What didn’t work? I run Ollama on my M2 Pro and all models work.

I'm getting "Error: llama runner process has terminated" when trying to run the model. According to this ticket[1], it's a memory issue. Not sure why 16GB ram are struggling with a 7b model, though.

[1] https://github.com/jmorganca/ollama/issues/1231

Re: Ask HN: What's the best hardware to run small/medium models locally?

#68

Post by wenc »

addandsubtract wrote: post #67
wenc wrote: post #62

Earlier quoted context omitted.

What didn’t work? I run Ollama on my M2 Pro and all models work.

I'm getting "Error: llama runner process has terminated" when trying to run the model. According to this ticket[1], it's a memory issue. Not sure why 16GB ram are struggling with a 7b model, though. [1] https://github.com/jmorganca/ollama/issues/1231

Could you try another model like Mistral 7b and see if triggers the same error? (And check your available memory)

I have 16GB too and have no trouble running even 13B models.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#69

Post by bryancoxwell »

spacecadet wrote: post #55
zer00eyz wrote: post #41

1. The GPU market is a mess! https://www.tweaktown.com/news/94394/amds-top-end-rdna-3-sal... Insiders who watch the prices and talk to VAR's all say that the channels seem stuffed and that prices are holding back sales. 2. AMD: They may change the land scape in coming months. And it looks like the US gov restrictions on GPU's are going to impact price in the server market in 2024. 3. The stacks are evolving quickly.…

This is what I did. 3080 eventually became a 3090 off ebay and 32gb now 128... but all on a budget and over time. Also as other have pointed out, it depends... I run models on Raspberry Pi's as well, one is doing live network detection...

Have you written any blog posts or anything about the one you have doing live network detection? I’d be really curious to hear more.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#70

Post by bee_rider »

webnrrd2k wrote: post #32
bee_rider wrote: post #21

Earlier quoted context omitted.

Have there been issues with CUDA on Linux? Having their cards run in servers is a big part of their business model, and Linux owns that market, so I’m surprised if their support is getting worse. Things like poor Wayland support—sure. But then, why would somebody use this matrix-multiplication accelerator to draw graphics, right?

On my home system I've been running 2x RTX 2070's on Fedora and have had serious enough problems. It's been fairly stable for a while, but the last week or so I keep having the screen go black and not come back. I'm going to try Debian as it's supposed to have better support for nvidia cards. I've been using Fedora or Redhat for a long time, and I'd rather not switch, but these driver issues make the system unusable.

So one is for graphics and the other is for CUDA or something?
Post reply on HN