Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

821–830 of 1001 posts

Re: Apple M3 Ultra

#821

Post by tgma »

lynguist wrote: post #718
lhl wrote: post #560

Earlier quoted context omitted.

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

Pretty sure this has absolutely nothing to do with Deepseek and even local LLM at large, which has been a thing for a while and an obvious use case original Llama leak and llama.cpp coming around.

Fact is Mac Pros in the Intel days supported 1.5TB RAM in some configurations[1] and that was 6 years ago expectations of their high end customer base. They needed to address the gap for those customers so they would have shipped such a product regardless. Local LLM is cherry-on-top. Deepseek in particular almost certainly had nothing to do with it. They will still need to double their supported RAM in their SoC to get there. Perhaps in a Mac Pro or a different quad-Max-glued chip.

[1]: https://support.apple.com/en-us/101639

Re: Apple M3 Ultra

#822

Post by forrestthewoods »

reitzensteinm wrote: post #813
forrestthewoods wrote: post #738

Earlier quoted context omitted.

I don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.

Chip? Yes. Product? Not necessarily... It's not completely out of the question that the 512gb version of M3 Ultra was built for their internal Apple silicon servers powering Private Compute Cloud, but not intended for consumer release, until a compelling use case suddenly arrived. I don't _think_ this is what happened, but I wouldn't go as far as to call it impossible.

DeepSeek R1 came out Jan 20.

Literally impossible.

Re: Apple M3 Ultra

#823

Post by RachelF »

wtallis wrote: post #786
hot_gril wrote: post #696

Earlier quoted context omitted.

M1 came out before the LLM rush, though

The M1 is in a product segment where discrete GPUs have been gone for decades , in favor of integrated graphics that shares one pool of RAM with the CPU. The better question to ask is why Apple kept using that unified memory design even when moving up to larger chips like the M1 Max and M1 Ultra.

I'd guess that they inherited it from the iPhone chips. It was nice and fast and also makes Apple a lot of profit as no third party RAM is possible.

Re: Apple M3 Ultra

#824

Post by tgma »

bustling-noose wrote: post #778
forrestthewoods wrote: post #738

Earlier quoted context omitted.

I don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.

My thoughts too. This product was in the pipeline maybe 2-3 years ago. Maybe with LLMs getting popular a year ago they tried to fit more memory but it’s almost impossible to do that that close to a launch. Especially when memory is fused not just a module you can swap.

Your conclusion is correct but to be clear the memory is not "fused." It's soldered close to the main processor. Not even a Package-on-Package (two story) configuration.

See photo without heatspreader here: https://wccftech.com/apple-m2-ultra-soc-delidded-package-siz...

Re: Apple M3 Ultra

#825

Post by jms55 »

johntitorjr wrote: post #40

Lots of AI HW is focused on RAM (512GB!). I have a cost-sensitive application that needs speed (300+ TOPS), but only 1GB of RAM. Are there any HW companies focused on that space?

Like others have said, basically traditional GPUs (RTX 40/50 series in particular, 20/30 series have much weaker tensor cores).

In terms of software, recent NVIDIA and AMD research has focused on fast evaluation of small ~4 layer MLPs using FP8 weights for things like denoising, upscaling, radiance caching, and texture and material BRDF compression/decompression.

NVIDIA has just put out some new graphics API extensions and samples/demos for loading a chunk of neural net weights and performing inference from within a shader.

Re: Apple M3 Ultra

#826

Post by pklausler »

musicale wrote: post #817
pklausler wrote: post #449

Earlier quoted context omitted.

This prompts an "old guy anecdote"; forgive me. When I was much younger, I got to work on compilers at Cray Computer Corp., which was trying to bring the Cray-3 to market. (This was basically a 16-CPU Cray-2 implemented with GaAs parts; it never worked reliably.) Back then, HPC performance was measured in mere megaflops. And although the Cray-2 had peak performance of nearly 500MF/s/CPU, it was really hard to attain,…

> so you had to have lots of operand re-use to not be memory-bound Looking at Nvidia's spec sheet, an H100 SXM can do 989 tf32 teraflops (or 67 non-tensor core fp32 teraflops?) and 3.35 TB/s memory (HBM) bandwidth, so ... similar problem?

There is caching today.

Re: Apple M3 Ultra

#827

Post by reitzensteinm »

forrestthewoods wrote: post #822
reitzensteinm wrote: post #813

Earlier quoted context omitted.

Chip? Yes. Product? Not necessarily... It's not completely out of the question that the 512gb version of M3 Ultra was built for their internal Apple silicon servers powering Private Compute Cloud, but not intended for consumer release, until a compelling use case suddenly arrived. I don't _think_ this is what happened, but I wouldn't go as far as to call it impossible.

DeepSeek R1 came out Jan 20. Literally impossible.

The scenario is that the 512gb M3 Ultra was validated for the Mac Studio, and in volume production for their servers, but a business decision was made to not offer more than a 256gb SKU for Mac Studio.

I don't think this happened, but it's absolutely not "literally impossible". Engineering takes time, artificial segmentation can be changed much more quickly.

Re: Apple M3 Ultra

#829

Post by vaxman »

nightski wrote: post #794
lynguist wrote: post #718

Earlier quoted context omitted.

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

$10k to run a 4 bit quantized model. Ouch.

[flagged]

Re: Apple M3 Ultra

#830

Post by [deleted] »

kridsdale1 wrote: post #605
bob1029 wrote: post #570

Earlier quoted context omitted.

> The question is if a llm will run with usable performance at that scale? For the self-attention mechanism, memory bandwidth requirements scale ~quadratically with the sequence length.

Someone has got to be working on a better method than that. Hundreds of billions are at stake.

[deleted]
Post reply on HN