I’ve been running Orca 2 13B on M1 Pro with 32GB of RAM with LLM Studio and GPU acceleration quite nicely. https://huggingface.co/TheBloke/Orca-2-13B-GGUF
Ollama runs 13b models just fine on my M2 Air with 16 GB
Ask HN: What's the best hardware to run small/medium models locally?
51–60 of 99 posts
Re: Ask HN: What's the best hardware to run small/medium models locally?
#52I'm using a DUO 16 2023 with 4090 16GB and ryzen 9 7945HX (16c/32t). It also uses another 32 GB shared RAM which makes it a 48GB 4090. It's quite a bit slower than a full on 4090 but it can load decent sized models and works well.
Tested both Linux (some things will need manual patching) and windows. Works like a charm.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#53Earlier quoted context omitted.
Yeah, the 3090 is a meme in local AI communities. Additionally, the support is amazing because its essentially the same architecture as an A100. The 3060 is popular too, being a 3090 cut in half.
> Yeah, the 3090 is a meme in local AI communities. Calling the 3090 a "meme" makes it sound like the 3090 is a joke. Do you mean that the 3090 is "well-known" in local AI communities?
Yeah, I just meant that is like the only option, which is crazy because its a 2020 GPU.
There are lots of questions about what hardware to get for ML, and the generic answer is basically always "get a 3090." Its so frequently recommended that it feels like a meme to me.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#54MacBook, thanks to Apple's new MLX framework.
Which model can I use with 16GB ram? I tried Ollama with WizardCoder 7b, but it didn't work.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#551. The GPU market is a mess! https://www.tweaktown.com/news/94394/amds-top-end-rdna-3-sal... Insiders who watch the prices and talk to VAR's all say that the channels seem stuffed and that prices are holding back sales. 2. AMD: They may change the land scape in coming months. And it looks like the US gov restrictions on GPU's are going to impact price in the server market in 2024. 3. The stacks are evolving quickly.…
This is what I did. 3080 eventually became a 3090 off ebay and 32gb now 128... but all on a budget and over time.
Also as other have pointed out, it depends... I run models on Raspberry Pi's as well, one is doing live network detection...
Re: Ask HN: What's the best hardware to run small/medium models locally?
#56Re: Ask HN: What's the best hardware to run small/medium models locally?
#57Any links to setting up a ChatGPT-like experience that is entirely local - ie. no connectivity to the web/cloud?
https://github.com/lmstudio-ai - several models to choose, laso API access if you want
Re: Ask HN: What's the best hardware to run small/medium models locally?
#58Somewhat related; how to run an uncensored model locally? I run llamafile (llamafile-server-0.1-llava-v1.5-7b-q4 and mistral-7b-instruct-v0.1-Q4_K_M-server) ones on my macbook m1 and they run file (fast enough for playing), but they both seem neutered quite a bit. It's hard to get them off the rails and mistral (the above one) actually barfs really quickly just repeating the same letter (fffffff usually) where it sho…
Yes, I wonder the same. I'm not so much about porn but pretty much every conversation with a model gets abruptly cut off when it's just getting interesting. Every model I tried - mistral, codellama etc. - is terribly maimed in this respect.