Apple M3 Ultra • Hacker News Board

Re: Apple M3 Ultra

#781

Post by j45 » Thu, Mar 06, 2025, 2:18 AM UTC

a1o wrote: post #739

lynguist wrote: post #718

Earlier quoted context omitted.

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

Any ideas on power consumption? I wonder how much power would that use. It looks like it would be more efficient than everything else that currently exists.

Looks like up to 480W listed here

https://www.apple.com/mac-studio/specs/

Re: Apple M3 Ultra

#782

Post by bustling-noose » Thu, Mar 06, 2025, 2:22 AM UTC

I wonder if Apple needs to reconsider Xserve. While Apple probably have some kind of server infrastructure teams, making their own server infrastructure out of their own hardware and software sounds like something they could explore. The app ecosystem coupled with apples servers offered in the cloud or ones you could buy would be a very interesting service business they could get into. Apples App Store needs better apps given how much the hardware is capable of now especially with iPads using M chips. A cloud backed hardware and software service specially designed for the app ecosystem sounds very tempting.

The hardware has evolved faster than software at Apple. It’s usually the opposite with most tech companies where hardware is unable to keep up with software.

Re: Apple M3 Ultra

#783

Post by dcchambers » Thu, Mar 06, 2025, 2:24 AM UTC

_alex_ wrote: post #152

apple keeps talking about the Neural Engine. Does anything actually use it? Seems like all the current LLM and Stable Diffusion packages (including MLX) use the GPU.

Historically no, Ollama and the like have only used the CPU+GPU.

That said, there are efforts being made to use the NPU. See: https://github.com/Anemll/Anemll - you can now run small models directly on your Apple Silicon Mac's NPU.

It doesn't give better performance but it's massively more power efficient than using the GPU.

Re: Apple M3 Ultra

#784

Post by bustling-noose » Thu, Mar 06, 2025, 2:26 AM UTC

wewewedxfgdf wrote: post #660

Computers these days - the more appealing, exciting, cooler desirable, the higher the price, into the stratosphere. $9499 What ever happening to competition in computing? Computing hardware competition used to be cut throat, drop dead, knife fight, last man standing brutally competitive. Now it's just a massive gold rush cash grab.

The MacBook Air with M4 chip 16gb ram and an amazing display and camera is just 999$. During back to school that will happen soon it’s 899$ with free AirPods. That’s really great value given how good the hardware is.

Re: Apple M3 Ultra

#785

Post by School-Cotton » Thu, Mar 06, 2025, 2:29 AM UTC

DidYaWipe wrote: post #713

swivelmaster wrote: post #426

Earlier quoted context omitted.

You could always just open a few Chrome tabs…

[flagged]

I downvote all Reddit-style memes, jokes, reference humor, catchphrases, and so on. It’s low-effort content that doesn’t fit the vibe of HN and actively makes the site worse for its intended purpose.

Re: Apple M3 Ultra

#786

Post by wtallis » Thu, Mar 06, 2025, 2:45 AM UTC

hot_gril wrote: post #696

RachelF wrote: post #688

Earlier quoted context omitted.

Some possible groups of reasons: 1. Until recently RAM amount was something the end user liked to configure, so little market demand. 2. Technically, building such a large system on a chip or collection of chiplets was not possible. 3. RAM speed wasn't a bottleneck for most tasks, it was IO or CPU. LLMs changed this.

M1 came out before the LLM rush, though

The M1 is in a product segment where discrete GPUs have been gone for decades, in favor of integrated graphics that shares one pool of RAM with the CPU. The better question to ask is why Apple kept using that unified memory design even when moving up to larger chips like the M1 Max and M1 Ultra.

Re: Apple M3 Ultra

#787

Post by ballooney » Thu, Mar 06, 2025, 2:52 AM UTC

I'm from the dark ages and am interested in this for non-AI things like CFD. What is the state of SDK support for these chips? Is there a nice rust or C++ library that abstracts the hardware and lets you just do very big Matrix multiplications?

Re: Apple M3 Ultra

#788

Post by jltsiren » Thu, Mar 06, 2025, 2:52 AM UTC

kllrnohj wrote: post #513

matt-p wrote: post #453

Earlier quoted context omitted.

Not really like for like. The pricing isn't as insane as you'd think, 96 to 256GB is 1500 which isn't 'cheap' but, it could be worse. All in 5,500 gets you a ultra with 256GB memory, 28 cores, 60 GPU cores, 10Gb network - I think you'd be hard pushed to build a server for less.

5,500 easily gets me either vastly more CPU cores if I care more about that or a vastly faster GPU if I care more about that. Or for both a 9950x + 5090 (assuming you can actually find one in stock) is ~$3000 for the pair + motherboard, leaving a solid $2500 for whatever amount of RAM, storage, and networking you desire. The M3 strikes a very particular middle ground for AI of lots of RAM but a significantly slower G…

Consumer hardware is cheap, if 192 GB of RAM is enough for you. But if you want to go beyond that, the Mac Studio is very competitively priced. A minimal Threadripper workstation with 256 GB is ~$7400 from Puget Systems. If you increase the memory to 512 GB, the price goes up to ~$10900. Mostly because 128 GB modules are about as expensive as what Apple charges for RAM. A Threadripper Pro workstation can use cheaper 8x64 GB for the same capacity, but because the base system is more expensive, you'll end up paying ~$11600.

Re: Apple M3 Ultra

#789

Post by rafram » Thu, Mar 06, 2025, 2:55 AM UTC

iamtheworstdev wrote: post #728

ben-schaaf wrote: post #721

Earlier quoted context omitted.

Except that you're stuck with macOS, so there aren't any drivers for NVIDIA, AMD or Intel GPUs.

and that no one is developing games for MacOS.

(1) That’s obviously not actually true.

(2) “No one” is developing games for Linux either, but the Steam Deck works great. Why? Wine, which you can run on macOS too.

Re: Apple M3 Ultra

#790

Post by burnerthrow008 » Thu, Mar 06, 2025, 3:06 AM UTC

whimsicalism wrote: post #322

aurareturn wrote: post #305

Earlier quoted context omitted.

Probably some kind of local LLM server. 1TB of 1.6 TB/s memory if you link 2 together. $20k total. Half the price of a single Blackwell chip.

with a vanishingly small fraction of flops and a small fraction of memory bandwidth

> with a vanishingly small fraction of flops and a small fraction of memory bandwidth

Is it though?

Wikipedia says [1] an M3 Max can do 14 TFLOPS of FP32, so an M3 Ultra ought to do 28 TFLOPS. nVidia claims [2] a Blackwell GPU does 80 TFLOPs of FP32. So M3 Ultra is 1/3 the speed of a Blackwell.

Calling that "a vanishingly small fraction" seems like a bit of an exaggeration.

I mean, by that metric, a single Blackwell GPU only has "a vanishingly small fraction" of the memory of an M3 Ultra. And the M3 Ultra is only burning "a vanishingly small fraction" of a Blackwell's electrical power.

nVidia likes throwing around numbers like "20 petaFLOPs" for FP4, but that's not real floating point... it's just 1990's-vintage uLaw/aLaw integer math.

[1] https://en.wikipedia.org/wiki/Apple_silicon#Comparison_of_M-...

[2] https://resources.nvidia.com/en-us-blackwell-architecture/da...

Edit: Further, most (all?) of the TFLOPs numbers you see on nVidia datasheets for "Tensor FLOPs" have a little asterisk next to them saying they are "effective" TFLOPs using the sparsity feature, where half the elements of the matrix multiplication are zeroed.