Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple caught off guard by AI demand for Mac Mini and Mac Studio

macrumors.com

Post reply on HN
Print view View original

241–250 of 638 posts

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#241

Post by Xeoncross »

e28eta wrote: post #121
Xeoncross wrote: post #88

Earlier quoted context omitted.

You are mistaken. I'm running Qwen 3.7 28B 4bit (MLX) with a 200k context window and everything total is 32GB RSS. Is this the best? No. That's why I said the sweet spot. Getting from 16GB macs to 32GB is perhaps possible. Jumping to 64GB or 128GB as the default is simply unreasonable right now.

I have a similar machine, and briefly poked at running a local LLM, but got discouraged after a couple days. The quality, responsiveness, and impact on the rest of the system didn’t seem worth it to me. What sorts of things are you doing with the local LLM? Anything interactive? Should I take another look?

Yes, 15-30 t/sec is pretty slow for local models so I recommend running local LLM tasks overnight where (vs paid plans) there isn't a risk of chewing through your token budget from a rogue loop or sub-agent. Even if it takes hours, you're sleeping anyway so no concern. herdr + pi works great for this but there are lots of harnesses.

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#242

Post by ronfriedhaber »

alasdair_ wrote: post #172

There is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded. I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's z…

> Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes.

Modal significantly improves this. Highly recommend.

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#243

Post by NegativeLatency »

jtap wrote: post #234
alasdair_ wrote: post #172

There is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded. I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's z…

Same, but with vision models. Unfortunately, I might be at my limit locally. I have three models that I'm using to find and identify objects in pictures. The largest dataset and model now takes about 8 hours per epoch on my Mac M4 with 16G memory.

Yeah this was what got me to start doing short rentals of bigger gpus in the clouds, upload your parquet files and it takes a couple of hours for a thing that would have my mac at 100% for a couple of days

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#244

Post by wat10000 »

jmalicki wrote: post #230
mv4 wrote: post #218

Earlier quoted context omitted.

It's quite shocking to me how many experienced, tech-savvy people, who used to care about cookies and ad tracking - are now willingly sending their business strategies, highly confidential contracts, and intimate personal issues to a cloud provider because "it is only $0.0x per million tokens!".

> It's quite shocking to me how many experienced, tech-savvy people, who used to care about cookies and ad tracking - are now willingly sending their business strategies, highly confidential contracts, and intimate personal issues to a cloud provider because "it is only $0.0x per million tokens!". Because there are more privacy guarantees there, depending on the provider. "But what if they violate their contract!" is…

I'm not sure about the tin-foil-hattedness of worrying about them violating their contract. But that's by-the-by. It is definitely not tin-foil-hat to worry about the data being taken in a breach.

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#245

Post by jldugger »

caycep wrote: post #122
yardie wrote: post #70

Earlier quoted context omitted.

You should listen to the podcast Acquired, specifically Nvidia and then Jensen Huang. They basically lucked into AI. Some researcher was using Nvidia gaming cards, and reached out to them about questions on CUDA. That email eventually turned them into a trillion dollar question.

to their credit, there was a lot of work behind "luck". Jensen showed up in person in 2017 in NEURIPS and he and likely a lot of his top brass basically sat down and read the entire conference proceedings/abstracts; there was likely a lot of work behind the scenes to behind the ML research pivot.

And 2017 was _late_ in their pivot. They'd been active for much, much longer. Last winter break I sat down to watch every GTC keynote, going back to 2009[1]. Even then, he's talking about expanding to non-graphics workloads. Google's GPU paper[2] just slotted naturally into their existing narrative and were happy to support it. "fortune favors the prepared" as they say.

[1]: https://www.youtube.com/watch?v=fYuH2Kl_b98 [2]: https://scholar.google.com/citations?view_op=view_citation&h...

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#247

Post by NetMageSCW »

SR2Z wrote: post #164
Someone wrote: post #154

Earlier quoted context omitted.

> Banks, Biglaw, and the Pentagon all do it in the cloud. In _a_ cloud: their own virtual private cloud. They also have enough power to negotiate contracts with strong privacy provisions.

This kind of stuff is available off the shelf at any major cloud provider.

Guaranteed privacy is available off the shelf?

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#248

Post by porkpieshoe »

scosman wrote: post #228
SaltyBackendGuy wrote: post #23

Earlier quoted context omitted.

Maybe a bit of hindsight bias / the outside view here, but I feel like they're completely asleep if they didn't anticipate strong demand for this specific use case.

They were investing in ANE and Metal before everyone in consumer. Hardly asleep. They just underestimated the market size, as pretty much everyone did.

[dead]

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#249

Post by dghlsakjg »

Grombobulous wrote: post #19

I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have. I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me. Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardw…

I'm using a Mac to do bulk diarized transcription (STT). Most services run in the $.05-1.00+ per hour of transcribed audio.

My Mac can do ~200x realtime (1 hour takes 20s or so). I can do several thousand hours per day. Its pretty incredible

Not sure how much that qualifies as AI vs LLM usage, but it seems to work pretty good

Re: Apple caught off guard by AI demand for Mac Mini and Mac Studio

#250

Post by embedding-shape »

ronfriedhaber wrote: post #242
alasdair_ wrote: post #172

There is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded. I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's z…

> Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. Modal significantly improves this. Highly recommend.

Is Modal at all similar to Vast.ai or just related because "It's for AI"? I looked at Modal's page for training, and it talks about using some SDK and other junk, can you not just get a beefy instance from Modal with tons of VRAM to do what you want with?
Post reply on HN