Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

771–780 of 1001 posts

Re: Apple M3 Ultra

#771

Post by PaulHoule »

jauntywundrkind wrote: post #750
burnerthrow008 wrote: post #374

Earlier quoted context omitted.

I wonder if that’s something new, or just the same virtual network interface that’s been around since the TB1 days (a new network interface appears when you connect two Macs with a TB cable)

Its the same host-to-host usb network, I believe. I'm super interested in the clustering capability. At launch people said they were only getting like 11Gbps from their TB4 drive arrays, which was really way less than expected. Apple does kind of advertise that each TB port has its own controllers. Which gives me hope that whatever 1x port can do 6x can do 6x better. AMD's Strix Halo victory feels much more shallow t…

In 2017 I was working for a company that was trying to develop foundation models and I was developing a framework for training what were then large neural network [1] and other models.

It was "yet another mac-oriented startup" but I had them get me an Alienware laptop because I could get one with a 1070 mobile card that meant I could train on my laptop whereas the data sci's had to do everything on our DGX-1. [2]

Today it is the other way around, the Mac Studio looks like the best AI development workstation you can get.

[1] I was really partial to a character-level CNN model we had

[2] CEO presented next to Jensen Huang at a NVIDIA conference, his favorite word was "incredible". I thought it was "incredible" when I heard they got bought by Nike, but it was true.

Re: Apple M3 Ultra

#772

Post by aurareturn »

adrian_b wrote: post #644
aurareturn wrote: post #546

Earlier quoted context omitted.

Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU. Source on this? CPUs would be very compute constrained.

According to Apple, the GPU of M3 Ultra has 80 graphics cores, which should mean 10240 FP32 execution units, the same like an NVIDIA RTX 4080 Super. However Apple does not say anything about the GPU clock frequency, which I assume that it is significantly less than that of NVIDIA. In comparison, a dual-socket AMD Turin can have up to 12288 FP32 execution units, i.e. 20% more than an Apple GPU. Moreover, the clock fre…

No one is using FP64 for AI inference.

Re: Apple M3 Ultra

#773

Post by ryao »

a1o wrote: post #739
lynguist wrote: post #718

Earlier quoted context omitted.

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

Any ideas on power consumption? I wonder how much power would that use. It looks like it would be more efficient than everything else that currently exists.

The M2 Ultra Mac Pro could reach a maximum of 330W according to Apple:

https://support.apple.com/en-us/102839

I assume it is similar.

Re: Apple M3 Ultra

#775

Post by aurareturn »

KingOfCoders wrote: post #575
aurareturn wrote: post #373

Earlier quoted context omitted.

what's the point of 512GB RAM for LLMs on this Mac Studio if the speed is painfully slow? You can fit the entire Deepseek 671B q4 into this computer and get 41 tokens/s because it's an MoE model.

Your comments went from "40 tokens/s by my calculations" to "40 tokens/s" to "41 tokens/s" Is there a dice involved in "your calculations?"

41 was when I learned it has a little over 800B/s.

Doesn't matter. All theorized because no one has publicly tested one.

Re: Apple M3 Ultra

#777

Post by worthless-trash »

pklausler wrote: post #449
dheera wrote: post #400

Earlier quoted context omitted.

Apples to oranges. NVIDIA cards have an order of magnitude more horsepower for compute than this thing. A B100 has 8 TB/s of memory bandwidth, 10 times more than this. If NVIDIA made a card with 512GB of HBM I'd expect it to cost $150K. The compute and memory bandwidth of the M3 Ultra is more in-line with what you'd get from a Xeon or Epyc/Threadripper CPU on a server motherboard; it's just that the x86 "way" of doin…

This prompts an "old guy anecdote"; forgive me. When I was much younger, I got to work on compilers at Cray Computer Corp., which was trying to bring the Cray-3 to market. (This was basically a 16-CPU Cray-2 implemented with GaAs parts; it never worked reliably.) Back then, HPC performance was measured in mere megaflops. And although the Cray-2 had peak performance of nearly 500MF/s/CPU, it was really hard to attain,…

That is a great story. Please never hesitate to drop these in.

Do you have a blog?

Re: Apple M3 Ultra

#778

Post by bustling-noose »

forrestthewoods wrote: post #738
lynguist wrote: post #718

Earlier quoted context omitted.

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

I don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.

My thoughts too. This product was in the pipeline maybe 2-3 years ago. Maybe with LLMs getting popular a year ago they tried to fit more memory but it’s almost impossible to do that that close to a launch. Especially when memory is fused not just a module you can swap.

Re: Apple M3 Ultra

#779

Post by dcchambers »

hu3 wrote: post #668
wewewedxfgdf wrote: post #660

Computers these days - the more appealing, exciting, cooler desirable, the higher the price, into the stratosphere. $9499 What ever happening to competition in computing? Computing hardware competition used to be cut throat, drop dead, knife fight, last man standing brutally competitive. Now it's just a massive gold rush cash grab.

It doesn't even run Linux properly. Could cost half of that and it would still be uninteresting for my use cases. For AI, on-demand cloud processing is magnitudes better in speed and software compatibility anyway.

There are legitimate use cases for local LLMs.

For example, I'll happily feed my entire directory of private notes/diary entries into an LLM running offline on my laptop. I would never do that with someone else's LLM running in the cloud.

Re: Apple M3 Ultra

#780

Post by kmacdough »

lynguist wrote: post #718
lhl wrote: post #560

Earlier quoted context omitted.

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

That or it's the luckiest coincidence! In all seriousness, Apple is fairly consistent about not pushing specs that don't matter and >256GB is just unnecessary for most other common workloads. Factors like memory bandwidth, core count and consumption/heat would have higher impact.

That said, I doubt it was explicitly for R1, but rather based the industry a few years ago when GPT 3s 170B was SOTA, but the industry was still looking larger. "As much memory as possible" is the name of the game for AI in a way that's not true for other workloads. It may not be true for AI forever either.

Post reply on HN