Any links to setting up a ChatGPT-like experience that is entirely local - ie. no connectivity to the web/cloud?
Ask HN: What's the best hardware to run small/medium models locally?
31–40 of 99 posts
Re: Ask HN: What's the best hardware to run small/medium models locally?
#32Earlier quoted context omitted.
Unless you use linux, where the quality of Nvidia support continues to decline.
Have there been issues with CUDA on Linux? Having their cards run in servers is a big part of their business model, and Linux owns that market, so I’m surprised if their support is getting worse. Things like poor Wayland support—sure. But then, why would somebody use this matrix-multiplication accelerator to draw graphics, right?
On my home system I've been running 2x RTX 2070's on Fedora and have had serious enough problems. It's been fairly stable for a while, but the last week or so I keep having the screen go black and not come back. I'm going to try Debian as it's supposed to have better support for nvidia cards. I've been using Fedora or Redhat for a long time, and I'd rather not switch, but these driver issues make the system unusable.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#33A used RTX3090 of eBay is the most interesting budget option by far. If you have twice the cash, go for a new RTX4090 for rougly twice the performance. If you need more than 24GB vram, you want to get comfortable with sharding across a few of the 3090's, or spend a lott more on a 48, 80, 100 GB card. If you feel adventurous, you can go a non nvidia route, but expect a lott of friction and elbow grease at least for no…
Yeah, the 3090 is a meme in local AI communities. Additionally, the support is amazing because its essentially the same architecture as an A100.
The 3060 is popular too, being a 3090 cut in half.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#34A gaming desktop PC with Nvidia 3060 12GB or better. Upgrade the GPU first if you can afford it, prioritizing VRAM capacity and bandwidth. Nvidia GPU performance will blow any CPU including M3 out of the water and the software ecosystem pretty much assumes you are using Nvidia. Laptop GPUs are not equivalent to the desktop ones with the same number so don't be fooled. 8x 3090 (purchased used) is a popular configurati…
[deleted]
Re: Ask HN: What's the best hardware to run small/medium models locally?
#35Nvidia GPU's are really your only choice. There is no framework as mature as CUDA and nvidia has been making the fastest hardware for decades. They know their stuff when it comes to architecture, so its unlikely that the hot new thing will actually be able to compete.
Unless you use linux, where the quality of Nvidia support continues to decline.
> where the quality of Nvidia support continues to decline.
what are you talking about?
Re: Ask HN: What's the best hardware to run small/medium models locally?
#36A gaming desktop PC with Nvidia 3060 12GB or better. Upgrade the GPU first if you can afford it, prioritizing VRAM capacity and bandwidth. Nvidia GPU performance will blow any CPU including M3 out of the water and the software ecosystem pretty much assumes you are using Nvidia. Laptop GPUs are not equivalent to the desktop ones with the same number so don't be fooled. 8x 3090 (purchased used) is a popular configurati…
[deleted]
Not a dumb question, but maybe dumb to make a whole comment instead of taking two seconds to google it
https://www.techspot.com/review/2625-nvidia-rtx-4090-laptop-...
Re: Ask HN: What's the best hardware to run small/medium models locally?
#37I think there are a couple of basic questions need answered before we can find a good solution: 1) What are you trying to do? 2) What's your budget? Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question. When I was fac…
And a third question - what kind of machine are you looking for?
If you want a laptop, that's going to be a very different machine to a desktop.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#38I'm happy with my used 3090.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#39A used RTX3090 of eBay is the most interesting budget option by far. If you have twice the cash, go for a new RTX4090 for rougly twice the performance. If you need more than 24GB vram, you want to get comfortable with sharding across a few of the 3090's, or spend a lott more on a 48, 80, 100 GB card. If you feel adventurous, you can go a non nvidia route, but expect a lott of friction and elbow grease at least for no…
Yeah, the 3090 is a meme in local AI communities. Additionally, the support is amazing because its essentially the same architecture as an A100. The 3060 is popular too, being a 3090 cut in half.
> Yeah, the 3090 is a meme in local AI communities.
Calling the 3090 a "meme" makes it sound like the 3090 is a joke. Do you mean that the 3090 is "well-known" in local AI communities?
Re: Ask HN: What's the best hardware to run small/medium models locally?
#40Any links to setting up a ChatGPT-like experience that is entirely local - ie. no connectivity to the web/cloud?