Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

macrumors.com

Post reply on HN
Print view View original

311–320 of 327 posts

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#311

Post by jghn »

subarctic wrote: post #309
ttul wrote: post #269

Earlier quoted context omitted.

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

> Most useful LLM work is done in parallel I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate hav…

What about when claude code starts spawning sub-agents? it's pretty good about doing that now even if a user doesn't request it. Does this could as parallel inference?

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#312

Post by subarctic »

jghn wrote: post #311
subarctic wrote: post #309

Earlier quoted context omitted.

> Most useful LLM work is done in parallel I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate hav…

What about when claude code starts spawning sub-agents? it's pretty good about doing that now even if a user doesn't request it. Does this could as parallel inference?

I haven't seen it do it by itself that much so far. Even then, a lot of the time is spent waiting for some shell command to complete, which isn't inference. I guess being able to do parallel inference is like having multiple cores in a cpu, you can get by without it by just sharing that single core, and having it just gives you a performance boost under certain circumstances

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#313

Post by Danox »

Jcowell wrote: post #216
ChrisMarshallNY wrote: post #188

Earlier quoted context omitted.

Sadly, that is an outdated PoV. It has probably not been valid, since last century. It's just that Apple isn't really focused on software development professionals, and it's still fashionable to throw shade on them, so we hear a lot of kvetching about it, in communities like this.

I’m interested to know the ways that Apple doesn’t cater to Software devs the way other OS does. I hate developing on a windows/non-unix machine

That’s just excuses for example how does a relatively small company like Black Magic Design With one of the best programs in its class manage to support three operating systems MacOS, Windows OS and Linux OS simultaneously?

There are many larger software companies that can support all three operating systems, they just make up excuses/reasons as to why they can’t perform over the years.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#314

Post by jghn »

subarctic wrote: post #312
jghn wrote: post #311

Earlier quoted context omitted.

What about when claude code starts spawning sub-agents? it's pretty good about doing that now even if a user doesn't request it. Does this could as parallel inference?

I haven't seen it do it by itself that much so far. Even then, a lot of the time is spent waiting for some shell command to complete, which isn't inference. I guess being able to do parallel inference is like having multiple cores in a cpu, you can get by without it by just sharing that single core, and having it just gives you a performance boost under certain circumstances

Interesting. For me it seems to be much more common that even basic requests start spinning up sub-agents. But I also routinely ask it to do this so maybe it's stored this as a preference somewhere deep in its bowels.

And yes, I agree. I find the experience better less for speed and more for context management. But it's far from necessary.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#315

Post by thephyber »

throw1234567891 wrote: post #82
thephyber wrote: post #26

Earlier quoted context omitted.

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device. It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short. But Meta and xAI putting A TON of AI compute onto the mar…

512GB M3 Ultra is out of stock, not coming back, and there’s nothing like it on the consumer market. That’s the reason they go for so much.

Do you honestly think that’s families buying them?

My money is on small companies or affluent programmers experimenting with some new hobby / business model.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#316

Post by bigyabai »

mercutio2 wrote: post #302
bigyabai wrote: post #278

Earlier quoted context omitted.

Outperforms doing what? Inference is not a homogeneous workload, memory bandwidth correlates to decode speed and layer swapping but not necessarily inference speed overall. The other half of that equation is latency, predicated on prefill performance which needs a powerful GPU and ideally ALU-level optimization to build larger KV caches quickly. Even the M5 gets smoked in this department, the M5 Max has a 50% longer…

For agentic work, you just cache the prefill kv cache of the relevant system prompts. TTFT is a little slow (10s of seconds, oh no!) the first time you boot up a new harness. I keep hearing people make this claim that TTFT is a problem, and… it just isn’t, if you’re running oMLX. Folks in my camp keep saying this, and folks in your camp keep beating a drum we tell you isn’t resonating. Not sure why I keep bothering t…

It's really not that hard. If Apple Silicon was competitive for inference, then Apple would not be shelving them: https://9to5mac.com/2026/03/02/some-apple-ai-servers-are-rep...

They're not usable for deployment. They're perfectly fine for "enthusiast" low-end usage with 10-30B models, but the same goes for almost every dGPU made in the last 10 years. Your Mac Studio cannot run frontier LLMs at an interactive speed, even Apple has given up on using it as an inference backbone.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#317

Post by thephyber »

uejfiweun wrote: post #48
thephyber wrote: post #26

Earlier quoted context omitted.

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device. It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short. But Meta and xAI putting A TON of AI compute onto the mar…

What makes you think the supply crunch is short-term?

The underlying demand for massive RAM builds is coming from AI hyperscalers.

There are lots of signals that the sector has been overinvested and that corporate customers are pulling back on spending as the cost of the APIs is revealed.

Once the hyperscalers start struggling to bay their debts (it will happen, just a question of time), there will be a supply glut.

So the only question is: do we share the same definition of “short term”.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#318

Post by dehrmann »

nsagent wrote: post #264
_verandaguy wrote: post #250

Earlier quoted context omitted.

I know public opinion polling supports that, but the parts of my social circle which are outside of tech seem to be, at worst, apathetic (and at best enthusiastic, though that's not a big fraction). That said, I think it's a good thing that this sentiment is coming to the forefront.

I was actually surprised to hear my brother-in-law deride LLMs as being useless for areas he has expertise in when I visited for the 4th of July. He was complaining that he would ask how to perform a certain repair on a car, and the LLMs he tried (ChatGPT & Grok) would give him a long involved process and he'd ask why not do it this simpler way and it would say, oh you're right! He just found it gave bad advice and r…

There's a lot of Gell Mann amnesia going on with LLMs.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#319

Post by boredatoms »

throw1234567891 wrote: post #169
vkazanov wrote: post #145

Earlier quoted context omitted.

I think the REALLY cool thing about apple's shared memory implementation is the ultra-wide memory bus. Otherwise, AMD is quite close to what Apple has, and Strix Halo is honestly incredible. Not sure what RDMA brings to the table.

RDMA increases the inference performance by a significant percentage across devices connected via Thunderbolt 5.4x512 is like a 2TB machine.

Thunderbolt RDMA is slower and higher latency than if apple just gave us PCIe, where we could put a (old) connectX card in for infiniband

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#320

Post by QuercusMax »

amazingman wrote: post #279
_verandaguy wrote: post #236

Earlier quoted context omitted.

I'm perfectly happy with Apple not becoming an "everything we do is AI-centric" business. I'm fatigued by it all at this point. It's streamlining the interesting and fun parts out of my job (by practical necessity of use there), and if I used it half as much outside of work I'm sure it'd do the same there too.

> It's streamlining the interesting and fun parts out of my job Interesting. For me it's streamlining the tedious and attentionally taxing parts of my work tasks. I love solving problems, I don't particularly love shaving yaks.

If i'm gonna shave a yak, I'd prefer to do it with a set of powered clippers, not a pair of embroidery scissors.
Post reply on HN