Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best hardware to run small/medium models locally?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 99 posts

Re: Ask HN: What's the best hardware to run small/medium models locally?

#81

Post by brucethemoose2 »

phanimahesh wrote: post #64
brucethemoose2 wrote: post #53

Earlier quoted context omitted.

Yeah, I just meant that is like the only option, which is crazy because its a 2020 GPU. There are lots of questions about what hardware to get for ML, and the generic answer is basically always "get a 3090." Its so frequently recommended that it feels like a meme to me.

It hits a good sweet spot between price and power, and there's usually some easily available. Availability is a major factor imo.

Yeah, exactly. Gaudi boards might be good for local genai... if you could find any.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#82

Post by dr_kiszonka »

dvfjsdhgfv wrote: post #59
dr_kiszonka wrote: post #27

Earlier quoted context omitted.

Possibly the easiest way would be to run them in a notebook on Amazon SageMaker. https://aws.amazon.com/sagemaker/notebooks/

So you just start a notebook and then what?

You then download a model you want, say, Llama 2. Install a Python package to interact with the model (possibly `pip install llama-cpp-python`) and have fun.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#83

Post by webnrrd2k »

bee_rider wrote: post #70
webnrrd2k wrote: post #32

Earlier quoted context omitted.

On my home system I've been running 2x RTX 2070's on Fedora and have had serious enough problems. It's been fairly stable for a while, but the last week or so I keep having the screen go black and not come back. I'm going to try Debian as it's supposed to have better support for nvidia cards. I've been using Fedora or Redhat for a long time, and I'd rather not switch, but these driver issues make the system unusable.

So one is for graphics and the other is for CUDA or something?

Yes, the Debian setup is going to be just for cuda and ml. It's intended for cheap and easy experiments, focusing on ml for small systems.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#84

Post by bdavbdav »

wenc wrote: post #61
m1sta_ wrote: post #42

Earlier quoted context omitted.

I run mistral on an M2 air and it's broadly similar to chatgpt.

How? I have an M2 Pro and I run 7B and 13B models through Ollama and also LM Studio. Because there’s no CUDA, the speed is much slower than ChatGPT. The answers from 7B are also not at the same quality as ChatGPT. (Lots of mistakes and hallucinations)

Is metal working?

Re: Ask HN: What's the best hardware to run small/medium models locally?

#85

Post by m463 »

MPSimmons wrote: post #13

I think there are a couple of basic questions need answered before we can find a good solution: 1) What are you trying to do? 2) What's your budget? Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question. When I was fac…

I've noticed that llama 2 + llama.cpp doesn't seem to even use the GPU much. I tried a better gpu (more speed, more memory) and my inference speed didn't increase.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#87

Post by calamari4065 »

I run a 13B Q4 llama variant on my ten year old server with two Xeon E5-2670, 128GB of RAM, and no GPU

It runs at under 3 tokens per second. I usually just give it my prompt and go make a coffee or something. The server is in my basement, you can barely hear the fans screaming at all.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#88

Post by wenc »

bdavbdav wrote: post #84
wenc wrote: post #61

Earlier quoted context omitted.

How? I have an M2 Pro and I run 7B and 13B models through Ollama and also LM Studio. Because there’s no CUDA, the speed is much slower than ChatGPT. The answers from 7B are also not at the same quality as ChatGPT. (Lots of mistakes and hallucinations)

Is metal working?

Yes but it’s still no ChatGPT 4.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#89

Post by MPSimmons »

m463 wrote: post #85
MPSimmons wrote: post #13

I think there are a couple of basic questions need answered before we can find a good solution: 1) What are you trying to do? 2) What's your budget? Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question. When I was fac…

I've noticed that llama 2 + llama.cpp doesn't seem to even use the GPU much. I tried a better gpu (more speed, more memory) and my inference speed didn't increase.

Make sure that you're telling it to use the GPU. How are you launching llama_cpp?

Re: Ask HN: What's the best hardware to run small/medium models locally?

#90

Post by RandomWorker »

puchatek wrote: post #9
RandomWorker wrote: post #7

I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD. If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.

Can I ask what you're using it for?

No, that’s the whole point of running this stand alone without anyone spying.
Post reply on HN