Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best hardware to run small/medium models locally?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 99 posts

Re: Ask HN: What's the best hardware to run small/medium models locally?

#21

Post by bee_rider »

sam_lowry_ wrote: post #16
jocaal wrote: post #10

Nvidia GPU's are really your only choice. There is no framework as mature as CUDA and nvidia has been making the fastest hardware for decades. They know their stuff when it comes to architecture, so its unlikely that the hot new thing will actually be able to compete.

Unless you use linux, where the quality of Nvidia support continues to decline.

Have there been issues with CUDA on Linux?

Having their cards run in servers is a big part of their business model, and Linux owns that market, so I’m surprised if their support is getting worse.

Things like poor Wayland support—sure. But then, why would somebody use this matrix-multiplication accelerator to draw graphics, right?

Re: Ask HN: What's the best hardware to run small/medium models locally?

#22

Post by modeless »

A gaming desktop PC with Nvidia 3060 12GB or better. Upgrade the GPU first if you can afford it, prioritizing VRAM capacity and bandwidth. Nvidia GPU performance will blow any CPU including M3 out of the water and the software ecosystem pretty much assumes you are using Nvidia. Laptop GPUs are not equivalent to the desktop ones with the same number so don't be fooled. 8x 3090 (purchased used) is a popular configuration for people who have money and want to run the biggest models, but splitting models between GPUs requires extra work.

Personally I have 1x 4090 because I like gaming too, but it isn't really a big improvement over 3090 for ML unless you have a specific use for FP8, because VRAM capacity and bandwidth are very similar.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#23

Post by PeterStuer »

A used RTX3090 of eBay is the most interesting budget option by far.

If you have twice the cash, go for a new RTX4090 for rougly twice the performance.

If you need more than 24GB vram, you want to get comfortable with sharding across a few of the 3090's, or spend a lott more on a 48, 80, 100 GB card.

If you feel adventurous, you can go a non nvidia route, but expect a lott of friction and elbow grease at least for now.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#24

Post by gchadwick »

Have you considered running on a cloud machine instead? You can rent machines on https://vast.ai/ for under $1 an hour that should work for small/medium models (I've mostly been playing with stable diffusion so I don't know what you'd need for an LLM off hand).

Good GPUs and Apple hardware is pricey. Get a bit of automation setup with some cloud storage (e.g backblaze B2) and you can have a machine ready to run your personally fined tuned model rapidly with a CLI command or two.

There will be a break even point of course. Though a major advantage of renting is you can move easily as the tech does. You don't want to sink large amounts of money into a GPU only to find the next new hot open model needs more memory than you've got.

Re: Ask HN: What's the best hardware to run small/medium models locally?

#26

Post by godelski »

MPSimmons wrote: post #13

I think there are a couple of basic questions need answered before we can find a good solution: 1) What are you trying to do? 2) What's your budget? Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question. When I was fac…

> What are you trying to do?

This is by far the most important question. Because frankly, I can run LLaMA on my raspberry pi. It's slow as hell and not suited for any real time task but there are definitely operations where this would be an appropriate cost effective solution (preferably with actually a smaller distilled model).

There is no one size-fits all solution. The general advice is going to be a general mid tier graphics card but I assume that's information OP already has or could have found just as easily by typing this question into Google or any LLM. So if you (OP) want better advice, we got to have more information. The more detailed, the better (if this is a commercial application, then the answer is A100 because geforce cards are not allowed to be used for commercial environments, but no one's really going to stop you either). Ask vague question, get vague answers. But we will ask refining questions to help you ask better questions too :)

Re: Ask HN: What's the best hardware to run small/medium models locally?

#27

Post by dr_kiszonka »

pogue wrote: post #12
f0000 wrote: post #8

Just run them on AWS

Can you expand more on how that's done or point me in the direction to a guide?

Possibly the easiest way would be to run them in a notebook on Amazon SageMaker.

https://aws.amazon.com/sagemaker/notebooks/

Re: Ask HN: What's the best hardware to run small/medium models locally?

#30

Post by firtoz »

gchadwick wrote: post #24

Have you considered running on a cloud machine instead? You can rent machines on https://vast.ai/ for under $1 an hour that should work for small/medium models (I've mostly been playing with stable diffusion so I don't know what you'd need for an LLM off hand). Good GPUs and Apple hardware is pricey. Get a bit of automation setup with some cloud storage (e.g backblaze B2) and you can have a machine ready to run your…

I will link a few that I haven't used yet but seem promising:

- https://octoai.cloud/

- https://www.fal.ai/

- https://vast.ai/ (linked by gchadwick above)

- https://www.runpod.io/

- https://www.cerebrium.ai/

Post reply on HN