Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

571–580 of 1001 posts

Re: Apple M3 Ultra

#571

Post by christiangenco »

alok-g wrote: post #564

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…

IMO it's all about privacy. Perhaps also availability if the main LLM providers start pulling shenanigans but it seems like that's not going to be a huge problem with how many big players are in the space.

I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a client to connect to it and hey presto you've got a secure private LLM everybody in the company can use while keeping data private.

Re: Apple M3 Ultra

#573

Post by nomel »

lostmsu wrote: post #550
bastardoperator wrote: post #479

Earlier quoted context omitted.

It's fast enough for me to cancel monthly AI services on a mac mini m4 max.

Hm, the AI services over 5 years cost half of m4 max minimal configuration which can barely run severely lobotomized LLaMA 70B. And they provide significantly better models.

It's probably much worse than that, with the falling prices of compute.

Re: Apple M3 Ultra

#574

Post by snitty »

epolanski wrote: post #569

Can anybody ELI5 why aren't there multi gpu builds to run LLMs locally? It feels like one should be able to build a good machine for 3/4k if not less with 6 16GB mid level gaming GPUs.

Reddit's LocalLLama has a lot of these. 3090s are pretty popular for these purposes. But they're not trivial to build and run at home. Among other issues are that you're drawing >1kW for just the GPUs if you have four of them at 100% usage.

Re: Apple M3 Ultra

#575

Post by KingOfCoders »

aurareturn wrote: post #373
behnamoh wrote: post #35

819GB/s bandwidth... what's the point of 512GB RAM for LLMs on this Mac Studio if the speed is painfully slow? it's as if Apple doesn't want to compete with Nvidia... this is really disappointing in a Mac Studio. FYI: M2 Ultra already has 800GB/s bandwidth

what's the point of 512GB RAM for LLMs on this Mac Studio if the speed is painfully slow? You can fit the entire Deepseek 671B q4 into this computer and get 41 tokens/s because it's an MoE model.

Your comments went from

"40 tokens/s by my calculations"

to

"40 tokens/s"

to

"41 tokens/s"

Is there a dice involved in "your calculations?"

Re: Apple M3 Ultra

#576

Post by piotrpdev »

alok-g wrote: post #564

Two questions for the fellow HNers: 1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible? 2. Wit…

1. Lower latency for real time tasks e.g. transcription + translation?

Re: Apple M3 Ultra

#577

Post by [deleted] »

lauritz wrote: post #46

They update the Studio to M3 Ultra now, so M4 Ultra can presumably go directly into the Mac Pro at WWDC? Interesting timing. Maybe they'll change the form factor of the Mac Pro, too? Additionally, I would assume this is a very low-volume product, so it being on N3B isn't a dealbreaker. At the same time, these chips must be very expensive to make, so tying them with luxury-priced RAM makes some kind of sense.

[deleted]

Re: Apple M3 Ultra

#578

Post by hinkley »

InTheArena wrote: post #29

Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…

Given that the M1 Ultra and M2 Ultra also exist, I'd expect either straight binning, or two designs that use mostly the same designs for the cores but more of them and a few extra features.

I love Apple but they love to speak in half truths in product launches. Are they saying the M3 Ultra is their first Thunderbolt 5 computer? I don't recall seeing any previous announcements.

Re: Apple M3 Ultra

#579

Post by yxhuvud »

aurareturn wrote: post #365
hedora wrote: post #349

Earlier quoted context omitted.

That’s a laptop part, so it makes different tradeoffs. Somewhere on the internet there is a tdp wattage vs performance x-y plot. There’s a pareto optimal region where all the apple and amd parts live. Apple owns low tdp, AMD owns high tdp. They duke it out in the middle. Intel is nowhere close to the line. I’d guess someone has made one that includes datacenter ARM, but I’ve never seen it.

High TDP? You mean server-grade CPUs? Apple doesn't make those.

It also include gaming machines. Of course, Apple also don't make those.

Re: Apple M3 Ultra

#580

Post by crowcroft »

jsheard wrote: post #157
crowcroft wrote: post #128

Earlier quoted context omitted.

Agree with this, and it doesn't seem like it's a priority for Apple to bring the kind of expandability back any time soon. Maybe they can bring back the trash can.

Isn't the Mac Studio the new trash can? I can't think of how a non-expandable Mac Pro could be meaningfully different to the Studio unless they introduce an even bigger chip above the Ultra.

Oh yea I wasn't clear I just meant bring back the design - agree the studio basically is the trash can.
Post reply on HN