Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

151–160 of 207 posts

Re: My local model setup on an M4 Pro Mac Mini

#151

Post by thrw93747572007 »

usrnm wrote: post #86
hkchad wrote: post #5

Earlier quoted context omitted.

I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…

> It's free It isn't, the cost is included in your electricity bill, not even talking about the cost of your time to set it up. It's very possible that it costs you more than a cloud mode would, you just don't want to calculate it properly.

It sounds like they are doing something similar to what I described in my other post below. Personal media station.

That can be done on hardware that quite a lot of people basically just have and don't use 24/7 to the max - because it is their gaming machine or their programming and compiling workhorse, for example. Of course you are paying for additional electricity but even with napkin-math instead of a "proper" calculation, you are unlikely to pay more for running your own instead of something commercial (and that can be offset further with some of the "modern" electricity contracts and/or PV and battery storage). Especially if we are talking about a stack that runs most of/all the time when you are not using your machine and makes LLM calls regularly while running.

The work in software/admin to get whatever you want set up is similiar no matter which infrastructure you use.

Re: My local model setup on an M4 Pro Mac Mini

#153

Post by Youden »

ttul wrote: post #64

Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that? If there was a "Mullvad of GPU clouds", would that solve…

Chutes, Near AI, Phala and Tinfoil all offer various privacy assurances around inference. Some of the bigger providers also offer "zero data retention".

The problem I have with these is that the guarantees aren't strong enough (Phala, Near) or the models are old (Tinfoil). Chutes is mostly pretty good (cryptographic security all the way to the GPU) but I'm not sure it's possible to cryptographically verify the precise source code they run on the mode.

Re: My local model setup on an M4 Pro Mac Mini

#154

Post by t1E9mE7JTRjf »

Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux?

My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps.

Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people using mac minis for AI, to the point it almost became a meme.

Edit: yes I know this article is about local models, my question is a bit more general.

Re: My local model setup on an M4 Pro Mac Mini

#155

Post by bbatha »

t1E9mE7JTRjf wrote: post #154

Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux? My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps. Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people…

Unified memory is the answer, it’s one of the cheapest ways to get >64GB of gpu addressable memory.

Re: My local model setup on an M4 Pro Mac Mini

#156

Post by wuschel »

t1E9mE7JTRjf wrote: post #154

Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux? My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps. Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people…

The main difference total other laptops of non-Apple make is/was unified memory (graphic VRAM + RAM) architecture. No need for an extra dedicated graphics card to get 64+ GB VRAM.

Re: My local model setup on an M4 Pro Mac Mini

#157

Post by LeBit »

bel8 wrote: post #144
LeBit wrote: post #113

Earlier quoted context omitted.

I think he’s talking about the Mac Mini unified memory. 48G RAM is pretty useful if you want to run k8s locally for tests / exploration

true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires. you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.

I actually have a Mac Mini M4 Pro with 48G. I gave the k8s example because this is what I was doing with it.

Was because I am back to using Linux as my workstation.

My Mac Mini is now a headless server for llama.cpp.

So, you are right that for these workloads , I would not be using the Mac Mini for k8s AND llama.

Another thing going against using a Mac for Linux containers is that there are no solutions that I know that properly manages memory : memory is given to the Linux vm , but never fluctuates if the needs in the vm are less than the initial request.

I know Orb Stack does that but is it proprietary. I think UTM does it , but not sure I would use UTM instead of Lima, Colima , multipass , etc to run containers.

Re: My local model setup on an M4 Pro Mac Mini

#158

Post by JKCalhoun »

I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is.

I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in.

(I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)

Re: My local model setup on an M4 Pro Mac Mini

#159

Post by EagnaIonat »

whatsThisBtn4 wrote: post #120
EagnaIonat wrote: post #104

Earlier quoted context omitted.

> Agents require at least DeepSeek pro and even that is the minimum. The Granite 4.2 models which are just recently out, are optimized to handle agentic workflows. For local models, it's about using the right model for the right job.

Waste of time when I can pay $20 a month for sol.

versus $0 with local models.

There will always be a reason to run frontier models, but local models are well at levels that assist with stuff that don't need that level of complexity.

Re: My local model setup on an M4 Pro Mac Mini

#160

Post by jimbobthemighty »

I just use Gemini Pro which comes bundled with a Chromebook; when the 'free' year runs out I buy another, initiate the free years Gemini again and then put the 'as new' Chromebook on eBay and get most of my money back. So Gemini Pro costs me less than £2 a month. And for my needs (investment research) that works well and is a pretty cheap compromise.
Post reply on HN