Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best hardware to run small/medium models locally?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 99 posts

Re: Ask HN: What's the best hardware to run small/medium models locally?

#41

Post by zer00eyz »

1. The GPU market is a mess! https://www.tweaktown.com/news/94394/amds-top-end-rdna-3-sal... Insiders who watch the prices and talk to VAR's all say that the channels seem stuffed and that prices are holding back sales.

2. AMD: They may change the land scape in coming months. And it looks like the US gov restrictions on GPU's are going to impact price in the server market in 2024.

3. The stacks are evolving quickly. What you buy for today may be supersede by something tomorrow that means you should have spent more or could have spent less.

If you want to play: Ram, is what matters most. GPU ram and system ram (in that order). Get the best GPU you can (ram wise) under clock it and then add system memory if you can. Once you have a test bed that works for you, renting/cloud is a way to scale and play with bigger toys till you have a better sense of what you want and/or need.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#42

Post by m1sta_ »

RandomWorker wrote: post #7

I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD. If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.

I run mistral on an M2 air and it's broadly similar to chatgpt.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#43

Post by rubyn00bie »

sam_lowry_ wrote: post #16
jocaal wrote: post #10

Nvidia GPU's are really your only choice. There is no framework as mature as CUDA and nvidia has been making the fastest hardware for decades. They know their stuff when it comes to architecture, so its unlikely that the hot new thing will actually be able to compete.

Unless you use linux, where the quality of Nvidia support continues to decline.

CUDA works great on Linux. Full stop. If you’re having issues it’s because you’ve done something bizarre, like installing multiple versions of the driver. I promise you. I’ve been there and it was wholly my fault. Is it obvious or necessarily easy to fix? Nope. But that is problem with Linux and not the driver or CUDA.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#44

Post by mobiuscog »

I was interested in Stable Diffusion / images, and also text generation.

I started playing with ComfyUI and Ollama.

An M1 studio ultra would generate a 'base' 512x512 image in around 6 seconds, and ollama responses seemed easily 'quick enough'. Faster than I could read.

On an I7-3930K, purely CPU only, a similar image would take around 2.5 minutes, and ollama was painful, as I would be waiting for the next word.

Then I switched to a 3080ti, which I hadn't been using for gaming as it got stupidly hot and I regretted having it. Suddenly it was redeemed.

On the 3080ti, the same images come out in less than a second, and ollama generation is even faster. Sure, I'm limited to 7B models for text (the mac could go much higher) and there will be limits with image size/complexity, but this thing is so much faster than I expected, and hardly generates any heat/noise at the same time - completely different to gaming. This is all a simple install under linux (pop os in this case).

tl;dr - A linux PC with a high-end GPU is the best value by far unless you really need big models, in my experience.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#46

Post by jaimex2 »

jocaal wrote: post #10

Nvidia GPU's are really your only choice. There is no framework as mature as CUDA and nvidia has been making the fastest hardware for decades. They know their stuff when it comes to architecture, so its unlikely that the hot new thing will actually be able to compete.

+1

Also I run Linux full time, there's nothing better.

Apple side is still severely broken, you'll be fighting x86 shenanigans regularly.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#47

Post by sertbdfgbnfgsd »

isoprophlex wrote: post #17
pogue wrote: post #11

I don't want to derail the OP's question, but would the same kind of system to run an LLM on also be suitable for an image generator like Stable Diffusion or does it work through different methods?

I think so. A big nvidia gpu will run both.

Is a GPU the same as a graphics card?

Re: Ask HN: What's the best hardware to run small/medium models locally?

#48

Post by jaimex2 »

sam_lowry_ wrote: post #16
jocaal wrote: post #10

Nvidia GPU's are really your only choice. There is no framework as mature as CUDA and nvidia has been making the fastest hardware for decades. They know their stuff when it comes to architecture, so its unlikely that the hot new thing will actually be able to compete.

Unless you use linux, where the quality of Nvidia support continues to decline.

Linux support is great, what are you on about? You running Nouveau drivers or something lol

Re: Ask HN: What's the best hardware to run small/medium models locally?

#49

Post by jaimex2 »

coolspot wrote: post #20
sam_lowry_ wrote: post #16

Earlier quoted context omitted.

Unless you use linux, where the quality of Nvidia support continues to decline.

I am running 3090 on Linux Mint and 2060 on Rocky (RHEL 9) without any issues. Both CUDA and regular desktop use.

Running KDE on both 2060 RTX laptop and 3080 RTX desktop, flawless.
Post reply on HN