Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best hardware to run small/medium models locally?

news.ycombinator.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Ask HN: What's the best hardware to run small/medium models locally?

#91

Post by bee_rider »

webnrrd2k wrote: post #83
bee_rider wrote: post #70

Earlier quoted context omitted.

So one is for graphics and the other is for CUDA or something?

Yes, the Debian setup is going to be just for cuda and ml. It's intended for cheap and easy experiments, focusing on ml for small systems.

Hmm, I’m still confused sorry. To are you devoting both dGPUs to GPGPU stuff and using the iGPU for the desktop? Or is one dGPU doing double duty?

I wonder if it is worth trying the iGPU, if you haven’t already.

It would be a shame to distro-hop for a non-preference-based reason like this.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#92

Post by steve_adams_86 »

sparker72678 wrote: post #72
steve_adams_86 wrote: post #5

Somewhat related. I’ve got an M2 Max Mac Studio with 32GB of ram. Is there anything interesting I can do with it in terms of ML? What’s the scene like on moderately powered equipment like this?

Try https://ollama.ai Super easy to get setup and play around. Probably start with Llama2 13b or 7b, and then Mistral 7b. If you want a dead-simple chat GUI for interfacing, check out Ollamac.

Wow, just like that. Straight to the races. Thank you!

Kind of regretting not getting 64GB of ram now. I didn't think I'd need it, but... Here I am, wishing I could run some of these models.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#93

Post by PeterStuer »

brucethemoose2 wrote: post #33
PeterStuer wrote: post #23

A used RTX3090 of eBay is the most interesting budget option by far. If you have twice the cash, go for a new RTX4090 for rougly twice the performance. If you need more than 24GB vram, you want to get comfortable with sharding across a few of the 3090's, or spend a lott more on a 48, 80, 100 GB card. If you feel adventurous, you can go a non nvidia route, but expect a lott of friction and elbow grease at least for no…

Yeah, the 3090 is a meme in local AI communities. Additionally, the support is amazing because its essentially the same architecture as an A100. The 3060 is popular too, being a 3090 cut in half.

I myself wanted a new 4090 (2.000€), but my budget constrained me to a used 3090 from eBay (800€) which has served very well so far.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#94

Post by cyanydeez »

mobiuscog wrote: post #44

I was interested in Stable Diffusion / images, and also text generation. I started playing with ComfyUI and Ollama. An M1 studio ultra would generate a 'base' 512x512 image in around 6 seconds, and ollama responses seemed easily 'quick enough'. Faster than I could read. On an I7-3930K, purely CPU only, a similar image would take around 2.5 minutes, and ollama was painful, as I would be waiting for the next word. Then…

the synopsis is basically, Nvidia wins processing value but the new M* chips have more money for value

Re: Ask HN: What's the best hardware to run small/medium models locally?

#95

Post by m463 »

MPSimmons wrote: post #89
m463 wrote: post #85

Earlier quoted context omitted.

I've noticed that llama 2 + llama.cpp doesn't seem to even use the GPU much. I tried a better gpu (more speed, more memory) and my inference speed didn't increase.

Make sure that you're telling it to use the GPU. How are you launching llama_cpp?

I was using the command line. llama-7b

I did some investigating, and I found it doesn't start using the GPU unless you have a lot of input (such as a long prompt).

Re: Ask HN: What's the best hardware to run small/medium models locally?

#96

Post by RandomWorker »

evnc wrote: post #79
RandomWorker wrote: post #7

I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD. If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.

Have you looked into quantization? At 8-bit quantization, a 7B model requires ~7GB of RAM (plus a bit of overhead); at 4-bit, it would require around 3.5GB and fit entirely into the RAM you have. Quality of generation does degrade a bit the smaller you quantize, but not as much as you may think.

This is interesting; I've written how I set it up here; https://christiaanse.ca/posts/running_llm/

Re: Ask HN: What's the best hardware to run small/medium models locally?

#97

Post by webnrrd2k »

bee_rider wrote: post #91
webnrrd2k wrote: post #83

Earlier quoted context omitted.

Yes, the Debian setup is going to be just for cuda and ml. It's intended for cheap and easy experiments, focusing on ml for small systems.

Hmm, I’m still confused sorry. To are you devoting both dGPUs to GPGPU stuff and using the iGPU for the desktop? Or is one dGPU doing double duty? I wonder if it is worth trying the iGPU, if you haven’t already. It would be a shame to distro-hop for a non-preference-based reason like this.

The screen goes dark & unrecoverable during normal use, not while using ml tools, so I just assumed it was a problem with nvidia's drivers being generally disagreeable with Fedora.

You make a a good point, I've been having one card do double-duty as hdmi output and GPGPU. I'll try the motherboard's built-in hdmi and see how that goes.

I've been really busy with other stuff for a few weeks and haven't really thought about the best way to fix this. Thanks for the suggestion!

Re: Ask HN: What's the best hardware to run small/medium models locally?

#98

Post by seltzergenius »

bryancoxwell wrote: post #69
spacecadet wrote: post #55

Earlier quoted context omitted.

This is what I did. 3080 eventually became a 3090 off ebay and 32gb now 128... but all on a budget and over time. Also as other have pointed out, it depends... I run models on Raspberry Pi's as well, one is doing live network detection...

Have you written any blog posts or anything about the one you have doing live network detection? I’d be really curious to hear more.

[dead]

Re: Ask HN: What's the best hardware to run small/medium models locally?

#99

Post by vin047 »

qingcharles wrote: post #76
l72 wrote: post #75

To add to this, I have a laptop with 32G of RAM and am able to run some 7B models on CPU. But I'd like to work on some larger models. Are there any eGPUs that can aid in this?

I have a 12-year-old desktop with 24GB of RAM and no modern GPU and can run the 7B models, it's just no fun :(

I have a similarly old desktop with 24GB RAM and outdated GPU. What sort of token/second were you able to get with the 7B models?
Post reply on HN