Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

macrumors.com

Post reply on HN
Print view View original

271–280 of 327 posts

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#271

Post by varispeed »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

You can always buy multiple Macs. I think Apple's great differentiator could be making frontier class models local. I you want more threads, buy more Macs.

I don't want to run any workflows on someone else's computers.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#272

Post by timy2shoes »

bigyabai wrote: post #244
ingenieroariel wrote: post #233

Earlier quoted context omitted.

I wrote: "we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.)" April, 2023 We are both late and early. https://news.ycombinator.com/item?id=35527692

You should not buy a fully loaded Mac Studio for AI unless you absolutely NEED macOS. You will be wasting so much electricity idling on prefill while your GPU pulls 150-250w from the wall. Buy an Nvidia Spark, then whatever cheap Mac you want to use as a thin client. There's no reason to force Apple Silicon's round peg into a square hole like AI inference.

Benchmarking that I've seen shows that the M5 Max outperforms the DGX Spark, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1tfzsd6/m5_vs_d... or https://www.reddit.com/r/LocalLLaMA/comments/1tr7hzw/psa/. Seems to me that Apple is doing pretty well with local AI inference.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#273

Post by GeekyBear »

uejfiweun wrote: post #48
thephyber wrote: post #26

Earlier quoted context omitted.

I don’t think we should use current prices as landmarks for large scale demand. That Studio’s current prices is inflated because of a (presumably) short term supply crunch, not because the average user is willing to pay $24k for a home AI inference device. It assumes that RAM remains supply constrained and that none of the existing RAM contracts are cut short. But Meta and xAI putting A TON of AI compute onto the mar…

What makes you think the supply crunch is short-term?

DRAM doesn't require cutting edge Fabs, and Chinese manufacturers have finally caught up to the DDR5 standard.

> Multiple motherboard and PC component makers move forward with Chinese-made memory validation

https://www.pcgamer.com/hardware/memory/multiple-motherboard...

The only thing standing in the way of a major Chinese DDR5 ramp up is money.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#274

Post by anon373839 »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

> A Mac Mini can run one LLM inference thread at a time.

That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#275

Post by albrewer »

onion2k wrote: post #7

Running models on-device on a Mac is immensely annoying though. Figuring out what will work out of BF16, FP8, BF16+FP8, NVFP4, INT8, GGUF ... the list goes on ... is 'non-obvious' at best. Apple do little to support with tooling. There's MLX, but unless you're happy to transform a model to that format yourself you'll be lagging a long way behind. Apps like LMStudio, Ollama, Draw Things, etc do a great job of simplify…

What problems are you running into? Ive been running multiple qwen coder next agents locally and it works fine.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#276

Post by MisterBastahrd »

sajithdilshan wrote: post #68
JumpCrisscross wrote: post #64

Earlier quoted context omitted.

> cannot imagine a personal computing usage which can justify a 10k machine For me, the privacy pitch wins. I have a friend visiting, however, who spends like $2,400 with Anthropic every year. That's a solid ROI even if the thing becomes obsolete after a couple years. (I'm still on my 2020 MacBook Pro. I love it and will be sad when I have to replace it.)

> That's a solid ROI even if the thing becomes obsolete after a couple years. How can that be a solid return on investment? There's no model you can run locally to have frontier model level performance. Also who spends 2.4k yearly for personal AI usage, like what's the usecase? If your friend is spending that money for his business then it's not personal computing.

I have a friend who is a teacher and spends that much yearly for ChatGPT and Claude. It saves him many hours a year in time assembling documents and creating posters, flyers, and personalized learning materials for his students.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#277

Post by yieldcrv »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

I think local inference will be fast enough

There is so much happening in that scene, where tokens/sec double or 10x

So I could see the same hardware doing 20 tokens/sec on a large model suddenly doing 200 tokens/sec in the future, a better device in the future doing 500 tokens/sec, while having vision models baked in, audio models etc

Users wont consciously switch to local, they will just have it and use it

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#278

Post by bigyabai »

timy2shoes wrote: post #272
bigyabai wrote: post #244

Earlier quoted context omitted.

You should not buy a fully loaded Mac Studio for AI unless you absolutely NEED macOS. You will be wasting so much electricity idling on prefill while your GPU pulls 150-250w from the wall. Buy an Nvidia Spark, then whatever cheap Mac you want to use as a thin client. There's no reason to force Apple Silicon's round peg into a square hole like AI inference.

Benchmarking that I've seen shows that the M5 Max outperforms the DGX Spark, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1tfzsd6/m5_vs_d... or https://www.reddit.com/r/LocalLLaMA/comments/1tr7hzw/psa/ . Seems to me that Apple is doing pretty well with local AI inference.

Outperforms doing what? Inference is not a homogeneous workload, memory bandwidth correlates to decode speed and layer swapping but not necessarily inference speed overall.

The other half of that equation is latency, predicated on prefill performance which needs a powerful GPU and ideally ALU-level optimization to build larger KV caches quickly. Even the M5 gets smoked in this department, the M5 Max has a 50% longer TTFT on Qwen's 27b dense model at only 16k of context, which is a pretty typical starting context to use for agentic editing in normal apps like OpenCode/Claude Code: https://raw.githubusercontent.com/Osmantic/MMBT-Messy-Model-...

For agentic, 50-256k token on-device coding sessions, the Spark will be faster and consume less power running larger models. Without an external GPU (which Apple doesn't support), Apple Silicon will always be bottlenecked during prefill. Apple's failure to address this with their GPU architecture is a big reason why Apple Silicon viewed as a waste of time and money for professional datacenter deployment.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#279

Post by amazingman »

_verandaguy wrote: post #236
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

I'm perfectly happy with Apple not becoming an "everything we do is AI-centric" business. I'm fatigued by it all at this point. It's streamlining the interesting and fun parts out of my job (by practical necessity of use there), and if I used it half as much outside of work I'm sure it'd do the same there too.

> It's streamlining the interesting and fun parts out of my job

Interesting. For me it's streamlining the tedious and attentionally taxing parts of my work tasks. I love solving problems, I don't particularly love shaving yaks.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#280

Post by ttul »

anon373839 wrote: post #274
ttul wrote: post #269

Earlier quoted context omitted.

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

> A Mac Mini can run one LLM inference thread at a time. That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines.

Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in a datacenter serving 1,000 copies of myself? That's going to be more efficient.

Post reply on HN