Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

491–500 of 1001 posts

Re: Apple M3 Ultra

#491

Post by intrasight »

InTheArena wrote: post #29

Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…

It certainly is held back and that is unfortunate. But if you can run your workloads on this amazing machine, then that's a lot of compute for the buck.

I assume that there's a community of developers focusing on leveraging this hardware instead of complaining about the operating system.

Re: Apple M3 Ultra

#492

Post by doctorpangloss »

ein0p wrote: post #467
valine wrote: post #408

Earlier quoted context omitted.

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

What people who did not actually work with this stuff in practice don't realize is the above statement only holds for batch size 1, sequence size 1. For processing the prompt you will need to read all the weights (which isn't a problem, because prefill is compute-bound, which, in turn is a problem on a weak machine like this Mac or an "EPYC build" someone else mentioned). Even for inference, batch size greater than 1…

Sure, nuance.

This is why Apple makes so much fucking money: people will craft the wildest narratives about how they’re going to use this thing. It’s part of the aesthetics of spending $10,000. For every person who wants a solution to the problem of running a 400b+ parameter neural network, there are 19 who actually want an exciting experience of buying something, which is what Apple really makes. It has more in common with a Birkin bag than a server.

Re: Apple M3 Ultra

#493

Post by fetus8 »

bastardoperator wrote: post #479
radlad wrote: post #422

Earlier quoted context omitted.

It is much slower than nVidia, but for a lot of personal-use LLM scenarios, it's very workable. And it doesn't need to be anywhere near as fast considering it's really the only viable (affordable) option for private, local inference, besides building a server like this, which is no faster: https://news.ycombinator.com/item?id=42897205

It's fast enough for me to cancel monthly AI services on a mac mini m4 max.

How much RAM are you running on?

Re: Apple M3 Ultra

#494

Post by okanesen »

TheRealPomax wrote: post #477
FloatArtifact wrote: post #301

Earlier quoted context omitted.

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…

Yeah they did? The M4 has a max memory bandwidth of 546GBps, the M3 Ultra bumps that up to a max of 819GBps. (and the 512GB version is $4,000 more rather than $10,000 - that's still worth mocking, but it's nowhere near as much)

Not that dramatic of an increase actually - the M2 Max already had 400GB/s and M2 Ultra 800GB/s memory bandwidth, so the M3 Ultra's 819GB/s is just a modest bump. Though the M4's additional 146GB/s is indeed a more noticeable improvement.

Re: Apple M3 Ultra

#496

Post by egorfine »

duped wrote: post #258
InTheArena wrote: post #92

Earlier quoted context omitted.

No native docker support, no headless management options (enterprise strength), Limited QoS management, lack of robust python support (out of the box), interactive user focused security model.

> No native docker support Honest question: why do you want this in MacOS? Do you understand what docker does? (it's fundamentally a linux technology, unless you are asking for user namespaces and chroot w/o SIP on MacOS, but that doesn't make sense since the app sandbox exists). MacOS doesn't have the fundamental ecosystem problems that beget the need for docker. If the answer is "I want to run docker containers bec…

> why do you want this in MacOS?

I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive.

I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side.

My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact the ssh server is only available on good days and only after Remote Desktop logged in at least once. Launchd makes you wanna crave systemd. etc, etc.

So, about docker. I would absolutely love to run my app in a containerized environment on a Mac in order to not touch the main OS.

Re: Apple M3 Ultra

#497

Post by xp84 »

jsheard wrote: post #157
crowcroft wrote: post #128

Earlier quoted context omitted.

Agree with this, and it doesn't seem like it's a priority for Apple to bring the kind of expandability back any time soon. Maybe they can bring back the trash can.

Isn't the Mac Studio the new trash can? I can't think of how a non-expandable Mac Pro could be meaningfully different to the Studio unless they introduce an even bigger chip above the Ultra.

> Mac Studio the new trash can?

Indeed, and tbh it really commits even more to the non-expandability that the Trashcan's designers seemed to be going for. After all, the Trashcan at least had replaceable RAM and storage. The Mac Studio has proprietary storage modules for no reason aside from Apple's convenience/profits (and of course the 'integrated' RAM which I'll charitably assume was done for altruistic reasons because of how it's "shared.")

The difference is that today users are accepting modern Macs where they rejected the Trashcan. I think it's because Apple's practices have become more widespread anyway*, and certain parts of the strategy like the RAM thing at least have upsides. That, and the thermals are better because the Trashcan's thermal design was not fit for purpose.

* I was trying to fix a friend's nice Lenovo laptop recently -- it turned out to just have some bad RAM, but when we opened it up we found it was soldered :(

Re: Apple M3 Ultra

#498

Post by aurareturn »

anentropic wrote: post #424
_alex_ wrote: post #152

apple keeps talking about the Neural Engine. Does anything actually use it? Seems like all the current LLM and Stable Diffusion packages (including MLX) use the GPU.

Yeah I agree. The Neural Engine is useful for a bunch of Apple features, but seems weirdly useless for any LLM stuff... been wondering if they'd address it on any of these upcoming products. AI is so hype right now it seems odd that they have specialised processor that doesn't get used for the kind of AI people are doing. I can see in the latest release: > Mac Studio is a powerhouse for AI, capable of running large l…

On the iPhone, it runs on the NPU.

Re: Apple M3 Ultra

#500

Post by xp84 »

raydev wrote: post #488
xp84 wrote: post #476

Earlier quoted context omitted.

> both get the fastest single core performance AND all the cores I'm a major Apple skeptic myself, but hasn't there always been a tradeoff between "fastest single core" vs "lots of cores" (and thus best multicore)? For instance, I remember when you could buy an iMac with an i9 or whatever, with a higher clock speed and faster single core, or you could buy an iMac Pro with a Xeon with more cores, but the iMac (non-Pro…

> hasn't there always been a tradeoff between "fastest single core" vs "lots of cores" (and thus best multicore)? Not in the Apple Silicon line. The M2 Ultra has the same single core performance as the M2 Max and Pro. No benchmarks for the M3 Ultra yet but I'm guessing the same vs M3 Max and Pro.

Okay, good to know. Interesting change then.
Post reply on HN