M2 MacBook? A certain graphics card? Running inference on CPU on my old Thinkpad isn't fun.
Ask HN: What's the best hardware to run small/medium models locally?
1–10 of 99 posts
Re: Ask HN: What's the best hardware to run small/medium models locally?
#2M2? That's some cope.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#3In my experience, an nvidia card with the most memory you can get — that’s more important than speed, as models are tending to get bigger, and streaming models really hits speed.
I don’t have any Mac experience.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#4MacBook, thanks to Apple's new MLX framework.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#5Somewhat related. I’ve got an M2 Max Mac Studio with 32GB of ram. Is there anything interesting I can do with it in terms of ML? What’s the scene like on moderately powered equipment like this?
Re: Ask HN: What's the best hardware to run small/medium models locally?
#6A data point for you: 7B models at 5-bit quantization run quite comfortably under llama.cpp on the AMD Radeon RX 6700 XT, which has 12GB VRAM and was part of a lot of gaming PC builds around 2021-22.
I can’t give this as a recommendation - there are far more tools available for Nvidia GPUs, but larger VRAM is available on AMD GPUs at lower prices from what I can see.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#7I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD.
If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.
Re: Ask HN: What's the best hardware to run small/medium models locally?
#8Just run them on AWS
Re: Ask HN: What's the best hardware to run small/medium models locally?
#9I’m running mistral 7B on a M1 Mac 8GB just barely. It’s ask a question get a coffee type of thing. No idea how this works, as 32 bit floats require 4 bytes and with 7B it would need to be swapping with the SSD. If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.
Can I ask what you're using it for?
Re: Ask HN: What's the best hardware to run small/medium models locally?
#10Nvidia GPU's are really your only choice. There is no framework as mature as CUDA and nvidia has been making the fastest hardware for decades. They know their stuff when it comes to architecture, so its unlikely that the hot new thing will actually be able to compete.