Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

201–208 of 208 posts

Re: My local model setup on an M4 Pro Mac Mini

#201

Post by pwthornton »

argee wrote: post #3
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.

What kinds of tasks are you using this for?

Re: My local model setup on an M4 Pro Mac Mini

#202

Post by albrewer »

usrnm wrote: post #86
hkchad wrote: post #5

Earlier quoted context omitted.

I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…

> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.

> the cost is included in your electricity bill

Acting like an extra $20 on my electric bill is equivalent to a $200/mo subscription is... a take.

Re: My local model setup on an M4 Pro Mac Mini

#203

Post by exe34 »

usrnm wrote: post #86
hkchad wrote: post #5

Earlier quoted context omitted.

I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…

> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.

Free heating in winter.

Re: My local model setup on an M4 Pro Mac Mini

#204

Post by ActorNightly »

bambax wrote: post #195
ActorNightly wrote: post #190

Earlier quoted context omitted.

You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance. There are 2 main reasons for running local LLMS. 1. Process private data/work with uncensored models. 2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs. The thing that is critical for 2 is that a) you have…

> You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models I have a beefy Linux box with a 4090 but never took the time to set it up properly beyond simple testing; any tutorial you would recommend?

Just ollama if you are feeling lazy. With ollama, pull the model (start with https://ollama.com/library/gemma3:27b), and `ollama run gemma3:27b`

If you want to build agentic frameworks, use llama.cpp with its built in http server, and build the framework with python

Re: My local model setup on an M4 Pro Mac Mini

#205

Post by JKCalhoun »

maherbeg wrote: post #180
JKCalhoun wrote: post #158

I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is. I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in. (I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)

I've seen https://www.lucebox.com/ as an interesting option

I was hoping for a software "LLM in a Box".

Re: My local model setup on an M4 Pro Mac Mini

#206

Post by brettdav »

hkchad wrote: post #43
brettdav wrote: post #39

Earlier quoted context omitted.

Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.

What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.

Yes, that makes sense. Some of my models currently run in ollama while others require their own inference servers. I’m curious about custom inference servers in bifrost and the ability to orchestrate keeping some models warm in memory while evicting and cycling other models. All of which span different providers in bifrost (I think). Obviously I can get the mechanics from an agent, more wondering about any experience with something similar.

Re: My local model setup on an M4 Pro Mac Mini

#207

Post by manmal »

sunaookami wrote: post #196
manmal wrote: post #183

Earlier quoted context omitted.

I think I'd use Discord instead. Clankers are happy to set it all up for you.

Discord is even worse than Telegram in the privacy aspect though!?

Depends what actors you trust. They are equally bad but personally I‘m way more comfortable with the Discord story.

Re: My local model setup on an M4 Pro Mac Mini

#208

Post by a96 »

exe34 wrote: post #203
usrnm wrote: post #86

Earlier quoted context omitted.

> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.

Free heating in winter.

Though much less energy efficient than a heat pump.
Post reply on HN