Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

macrumors.com

Post reply on HN
Print view View original

301–310 of 327 posts

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#301

Post by jazzyjackson »

throw1234567891 wrote: post #82
thephyber wrote: post #26

Earlier quoted context omitted.

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device. It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short. But Meta and xAI putting A TON of AI compute onto the mar…

512GB M3 Ultra is out of stock, not coming back, and there’s nothing like it on the consumer market. That’s the reason they go for so much.

Yea but how many of them are selling at that price? Dozens?

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#302

Post by mercutio2 »

bigyabai wrote: post #278
timy2shoes wrote: post #272

Earlier quoted context omitted.

Benchmarking that I've seen shows that the M5 Max outperforms the DGX Spark, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1tfzsd6/m5_vs_d... or https://www.reddit.com/r/LocalLLaMA/comments/1tr7hzw/psa/ . Seems to me that Apple is doing pretty well with local AI inference.

Outperforms doing what? Inference is not a homogeneous workload, memory bandwidth correlates to decode speed and layer swapping but not necessarily inference speed overall. The other half of that equation is latency, predicated on prefill performance which needs a powerful GPU and ideally ALU-level optimization to build larger KV caches quickly. Even the M5 gets smoked in this department, the M5 Max has a 50% longer…

For agentic work, you just cache the prefill kv cache of the relevant system prompts. TTFT is a little slow (10s of seconds, oh no!) the first time you boot up a new harness.

I keep hearing people make this claim that TTFT is a problem, and… it just isn’t, if you’re running oMLX.

Folks in my camp keep saying this, and folks in your camp keep beating a drum we tell you isn’t resonating. Not sure why I keep bothering to argue; you can’t buy a high RAM Mac Studio like mine anymore.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#303

Post by dickersnoodle »

ttul wrote: post #280
anon373839 wrote: post #274

Earlier quoted context omitted.

> A Mac Mini can run one LLM inference thread at a time. That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines. Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in…

Why on earth would you need to do that for most things? Not everyone wants to be the next Coreweave. It sounds like you're killing a fly with a cannon to me.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#304

Post by dansquizsoft »

officeplant wrote: post #251
not-a-llm wrote: post #242

Earlier quoted context omitted.

DDR3 is from 10 years ago. DDR2 is from 15.

And? I just upgraded my 2008 Thinkpad R61i to 8GB of DDR2 a few months ago while I was also upgrading to a core2duo. DDR2 and DDR3 are still in active use by SBC manufacturers.

> I just upgraded my 2008 Thinkpad R61i to 8GB of DDR2 a few months ago while I was also upgrading to a core2duo

Uhhh… ok… good for you I guess…

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#305

Post by fragmede »

7speter wrote: post #293
varispeed wrote: post #271

Earlier quoted context omitted.

You can always buy multiple Macs. I think Apple's great differentiator could be making frontier class models local. I you want more threads, buy more Macs. I don't want to run any workflows on someone else's computers.

As far as I know, there isn’t an interface like nvlink that allows these macs to work in tandem; they would just send their data over ethernet, maybe thunderbolt/usb-c?

Why yes you can! https://www.jeffgeerling.com/blog/2025/15-tb-vram-on-mac-stu...

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#306

Post by fragmede »

stasomatic wrote: post #253
windex wrote: post #96

Earlier quoted context omitted.

Yup, for openclaw and APIs you dont need a big PC. I run something lightweight on the RPI4 8gb. Many people run local LLMs which is where a mac is useful. Frankly I dont think you can beat the value of an openrouter subscription and API calls.

If you or anyone else don’t mind, I have a have a question or 2. I use Claude Pro ($20/m) as a glorified search engine (no ads/SEO) plus simple hobbyist dev things (shell scripts, managing my Mac, apps etc. I also use it for tasks like - “search the web for top ten selling EVs, put them in a table” and then iterate - pivot tables, charts, additional research”. It could be cars, it could be broccoli. Code Work has fac…

You're not getting it on OpenRouter for anywhere near that cheap.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#307

Post by giancarlostoro »

habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Its a shame RAM prices are getting in the way of Apple. The 512 GB RAM Mac Studios would have been worthwhile.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#308

Post by 3abiton »

throw1234567891 wrote: post #82
thephyber wrote: post #26

Earlier quoted context omitted.

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device. It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short. But Meta and xAI putting A TON of AI compute onto the mar…

512GB M3 Ultra is out of stock, not coming back, and there’s nothing like it on the consumer market. That’s the reason they go for so much.

Funnily the current high end Mac Studio are not suited for current LLMs. M3 Ultra is "quite an old" chip for AI, despite its bandwidth. The issue for running local models (especially LLMs), you need few things to align really well: 1) compute power (affecting PP) 2) VRAM capacity (affecting model size you can load) 3) Bandwidth (somewhat affecting decoding speed).

The issue with the M3 chip is the compute performance, as it doesn't fit well the transformer architecture. This changed with the M5 (apple baked their own matmul into the chip), which would significantly speed up PP (and video/image generation btw), making the M5 Ultra significantly faster than the M3 Ultra and in practice much more usable. You can try to load Kimi or GLM on M3 Ultra, but it's not usable. Now the M5 Ultra is not out yet, but undoubtedly it will be a superior offering, and shilling 15k on 512GB version is actually reasonable (if it's every priced remotely around that tag).

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#309

Post by subarctic »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

> Most useful LLM work is done in parallel

I guess what I'm doing is not considered that useful then? I usually only have zero, one, or occasionally two things actively doing inference at a time, be it claude code sessions or one of the chatgpt/claude web interfaces, and i bet that's true for like 95% of people using llms. And anyway i bet even the hardcore people using a bunch of parallel agents would appreciate having access to local, private inference for some things.

You're obviously right though that cloud inference isn't going away anytime soon

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#310

Post by Danox »

sajithdilshan wrote: post #61
JumpCrisscross wrote: post #20

Apple could probably sell a machine starting at $10,000 if they architected it as the sole place one’s Private Cloud Compute [1] ran. It would need a path to a $2,500 machine, I think. But this is a niche I don’t think another consumer-facing brand could do like Apple. [1] https://security.apple.com/blog/private-cloud-compute/

But that would be more for enterprise usage. I cannot imagine a personal computing usage which can justify a 10k machine.

$10,000 divided by five divided by 12 is only $166.66 a month without interest over 5 years a Mac M2 Ultra or Mac M3 Ultra are easily usable computers over five years, so are most of the medium to high-end laptops that Apple sells for one person or locally in a small company.

It’s not necessarily out of reach or unusable over the course of time, the thing I keep hearing over and over is how usable many people find Mac’s very useable over the course of time, obviously software support plays a big part of that.

On top of that, AI models are getting more useful despite their smaller size. The future is a personal computer future, not a mainframe computer one. Yes, larger computers are useful, but most people will not be using that larger computers, which will be relegated to universities and larger companies.

Post reply on HN