Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

861–870 of 1001 posts

Re: Apple M3 Ultra

#861

Post by sgt »

adrian_b wrote: post #516
InTheArena wrote: post #100

Earlier quoted context omitted.

GPU accessible RAM.

In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher. Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000. Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple s…

Anecdotal but it seems like the big EPYC rigs are getting very low tokens per second, and not even consistent. They are strained, as opposed to e.g. M3 Ultra that can likely sustain 40-50 tokens/s based on previous stats.

I'd like to see some proper benchmarking on this though, but it looks like the Apple systems might just be extremely good value if you want to run the large DeepSeek model.

Re: Apple M3 Ultra

#862

Post by petepete »

cxie wrote: post #325
FloatArtifact wrote: post #301

Earlier quoted context omitted.

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…

Guess what? I'm on a mission to completely max out all 512GB of mem...maybe by running DeepSeek on it. Pure greed!

Give Cities Skylines 2 a try.

Re: Apple M3 Ultra

#864

Post by kergonath »

water9 wrote: post #843
saagarjha wrote: post #833

Earlier quoted context omitted.

The thing that people are excited about here is unified memory that the GPU can address. Mac Pro had discrete GPUs with their own memory.

Intel integrated graphics, technically also used unified memory with the standard dram

Those also have terrible performance and worse bandwidth. I am not sure they are really relevant, to be honest.

Re: Apple M3 Ultra

#865

Post by RataNova »

cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

It's a game changer for sure.... 512GB of unified memory really pushes the envelope, especially for running complex AI models locally. That said, the real test will be in how well the dual-chip design handles heat and power efficiency

Re: Apple M3 Ultra

#867

Post by jahewson »

reitzensteinm wrote: post #813
forrestthewoods wrote: post #738

Earlier quoted context omitted.

I don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.

Chip? Yes. Product? Not necessarily... It's not completely out of the question that the 512gb version of M3 Ultra was built for their internal Apple silicon servers powering Private Compute Cloud, but not intended for consumer release, until a compelling use case suddenly arrived. I don't _think_ this is what happened, but I wouldn't go as far as to call it impossible.

That's absurd. Fabing custom silicon is not something anybody does for a few thousand internal servers. The unit economics simply don't work. Plus Apple is using OpenAI to provide its larger models anyway, so the need never even existed.

Re: Apple M3 Ultra

#868

Post by tobyhinloopen »

MR4D wrote: post #819
ksec wrote: post #22

Previous model of M2 Ultra had max memory of 192GB. Or 128GB for Pro and some other M3 model, which I think is plenty for even 99.9% of professional task. They now bump it to 512GB . Along with insane price tag of $9499 for 512GB Mac Studio. I am pretty sure this is some AI Gold rush.

Remember, that RAM is also VRAM, so 1/2 terabyte of VRAM ain’t cheap. By comparison, Apple is a downright bargain!

It doesn't have the bandwidth of dedicated GPU VRAM.

Re: Apple M3 Ultra

#869

Post by tobyhinloopen »

Sharlin wrote: post #329
dwighttk wrote: post #32

Earlier quoted context omitted.

Maybe .1% of tasks need this RAM, why are they charging so much?

It enables the use of giant AI models on a personal computer. Might not run too fast though. But at least it's possible at all .

What is stopping us from running these models on a PC with 512GB RAM?

Re: Apple M3 Ultra

#870

Post by jjuliano »

Currently, Docker does not support Metal GPUs.

When running LLMs on Docker with an Apple M3 or M4 chip, they will operate in CPU mode regardless of the chip's class, as Docker only supports Nvidia and Radeon GPUs.

If you're developing LLMs on Docker, consider getting a Framework laptop with an Nvidia or Radeon GPU instead.

Source: I develop an AI agent framework that runs LLMs inside Docker on an M3 Max (https://kdeps.com).

Post reply on HN