Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

macrumors.com

Post reply on HN
Print view View original

291–300 of 327 posts

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#291

Post by wolvoleo »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

> 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally

Well just because they don't sell them. Doesn't mean that will be the case forever.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#292

Post by Danox »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

You lost it at when you said forever…

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#293

Post by 7speter »

varispeed wrote: post #271
ttul wrote: post #269

Earlier quoted context omitted.

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

You can always buy multiple Macs. I think Apple's great differentiator could be making frontier class models local. I you want more threads, buy more Macs. I don't want to run any workflows on someone else's computers.

As far as I know, there isn’t an interface like nvlink that allows these macs to work in tandem; they would just send their data over ethernet, maybe thunderbolt/usb-c?

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#294

Post by Danox »

HPsquared wrote: post #125
AussieWog93 wrote: post #62

Earlier quoted context omitted.

I heard that China was spinning up DDR5 (but not HBM?) production in the next couple of years, with the hope of outcompeting Korea and Taiwan in the mid to long term.

It does seem like an opportunity on a silver platter for Chinese newcomers. Huge demand at the moment.

In two to four years, the Chinese will have at least half the memory market worldwide, and once in, they will continue forward and not look back.

I also believe there would be one or two tech companies that will get into memory by taking it in-house to make sure that they won’t have this problem again in the future.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#295

Post by Danox »

runako wrote: post #224
tl wrote: post #209

Earlier quoted context omitted.

I asked an Apple (via a sales rep who visited our company to showcase in internal iPad healthcare app) to please do this for iCloud when iCloud Drive was in-development. We would have easily paid $50,000 for a rack-able Mac Pro you could point "managed" devices at. Apple simply cannot comprehend the ask.

They had Xserve, it didn't sell well enough to survive.

How do you design XServe being tied to IBM/Intel the way you want to make it, and why would you, when you’re having problems with Intel, why? would you go any further with Intel hardware wise? Makes no sense. The next four years will be interesting with an engineer CEO in charge.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#296

Post by Danox »

egorfine wrote: post #215
ChrisMarshallNY wrote: post #208

Earlier quoted context omitted.

Good on ya. I'm not interested in fighting about this stuff. I've had people hating on me for using Apple since the 1980s. It gets a bit old. Sort of like high school popularity contests.

I'm an exclusive Apple user for a couple decades so I'm not fighting here :) just acknowledging the fact that Apple has some deficiencies

All companies have deficiencies in the tech world. However, I would rather be in Apple’s position today, in comparison to Intel or Microsoft, particularly over the last 20 years.

And I definitely feel the pain with Aperture, not making it to Apple Silicon.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#297

Post by anon373839 »

ttul wrote: post #280
anon373839 wrote: post #274

Earlier quoted context omitted.

> A Mac Mini can run one LLM inference thread at a time. That’s not accurate. With MLX, at least, parallel inference is both possible and useful. Model serving tools like LM Studio and oMLX support parallel generation with continuous batching, and the total throughput increases with it.

You are not wrong, but the practical reality of local hardware is to be batch-constrained in comparison with a multi-user inference cloud. You will never be able to compete cost effectively in your home lab with a cloud that has >100M end users streaming millions of inference requests per second across a gigantic fleet of machines. Can I run a few inferences in parallel on my Mac Mini? Yes. But put 1,000 Mac Minis in…

Oh certainly, a local machine can't parallelize like the cloud can. Does anyone think that, though? What it can do is provide good-enough inference to support common business needs, without creating unnecessary data risks.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#298

Post by toofy »

ttul wrote: post #269
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

Here’s the two main reasons why local inference won’t compete any time soon with the cloud: 1. Most useful LLM work is done in parallel. A Mac Mini can run one LLM inference thread at a time. The cloud can spool up dozens and spread that inference across efficiently batched operations over a fleet of hardware. 2. Faster inference hardware such as the chips from Cerebras and Groq cannot be run locally. But the advanta…

and a toyota camry can’t compete with a lamborghini but the camry vastly outsells them.

you’re far far far more likely to see a camry or equivalent in an americans driveway than you are a super car.

you’re also more likely to see an enthusiast with a corvette or equivalent muscle car spending way more than it’s worth to tinker on that car in their garage than you are a super car.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#299

Post by eastbound »

ceroxylon wrote: post #254
habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

I don't mind the subtle ML integrations that they have put in the photos app: plant ID, recognizing faces, removing background, OCR text search (even for handwriting!), etc.

You’re listing every feature I’d like to remove; Meanwhile Android users have great holiday pictures at the bottom of the Pyramids because they can remove the people from their pictures.

Re: Apple Silicon Exec Explains Mac Mini AI Demand and On-Device Future

#300

Post by rk06 »

habosa wrote: post #219

Apple has totally failed to deliver interesting AI experiences so far ... and I still think they're going to be the dominant provider of AI in 5 years. We're just one or two advances in chips / models / both away from being able to run very good local models for free on mid-tier Apple devices. The privacy, cost, and latency story there will be too much for OpenAI/Anthropic/Google to beat. Just writing this down so I…

There is common wisdom: during gold rush, sell shovels.

Apple's shovel (ahem, Mac mini) is the highest quality.with Companies burning money left, right and center, Apple can dispense with advertising altogether

Post reply on HN