My local model setup on an M4 Pro Mac Mini

Post by chrisweekly » Wed, Sep 02, 2026, 12:41 PM UTC

LeBit wrote: post #157

bel8 wrote: post #144

Earlier quoted context omitted.

true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires. you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.

I actually have a Mac Mini M4 Pro with 48G. I gave the k8s example because this is what I was doing with it. Was because I am back to using Linux as my workstation. My Mac Mini is now a headless server for llama.cpp. So, you are right that for these workloads , I would not be using the Mac Mini for k8s AND llama. Another thing going against using a Mac for Linux containers is that there are no solutions that I know t…

Post by VCFundedGenYer » Wed, Sep 02, 2026, 12:43 PM UTC

jimbobthemighty wrote: post #160

I just use Gemini Pro which comes bundled with a Chromebook; when the 'free' year runs out I buy another, initiate the free years Gemini again and then put the 'as new' Chromebook on eBay and get most of my money back. So Gemini Pro costs me less than £2 a month. And for my needs (investment research) that works well and is a pretty cheap compromise.

Post by neilwilson » Wed, Sep 02, 2026, 1:06 PM UTC

ux266478 wrote: post #54

Aurornis wrote: post #33

Earlier quoted context omitted.

> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars. > It does mean 8 multi-trillion parameter models unquantized runnin…

> Pretty expensive is an understatement. [...] If you could it would be multiple hundreds of thousands of dollars. Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and imp…

Post by phazonoverload » Wed, Sep 02, 2026, 1:20 PM UTC

JKCalhoun wrote: post #158

I like the in-depth description. Everything from the naming convention of the models (and how much RAM they require) as well as all the components needed underscores just how complicated this all still is. I suppose I am waiting for AI-in-a-Box to come along so I can (painlessly) join in. (I'm sure wrangling with all these esoteric aspects of LLMs though is fun for some people.)

Post by phazonoverload » Wed, Sep 02, 2026, 1:25 PM UTC

crossroadsguy wrote: post #37

> oMLX Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?