Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

511–520 of 1001 posts

Re: Apple M3 Ultra

#511

Post by staticman2 »

bastardoperator wrote: post #479
radlad wrote: post #422

Earlier quoted context omitted.

It is much slower than nVidia, but for a lot of personal-use LLM scenarios, it's very workable. And it doesn't need to be anywhere near as fast considering it's really the only viable (affordable) option for private, local inference, besides building a server like this, which is no faster: https://news.ycombinator.com/item?id=42897205

It's fast enough for me to cancel monthly AI services on a mac mini m4 max.

Smaller, dumber models are faster than bigger, slower ones.

What model do you find fast enough and smart enough?

Re: Apple M3 Ultra

#512

Post by choilive »

okanesen wrote: post #494
TheRealPomax wrote: post #477

Earlier quoted context omitted.

Yeah they did? The M4 has a max memory bandwidth of 546GBps, the M3 Ultra bumps that up to a max of 819GBps. (and the 512GB version is $4,000 more rather than $10,000 - that's still worth mocking, but it's nowhere near as much)

Not that dramatic of an increase actually - the M2 Max already had 400GB/s and M2 Ultra 800GB/s memory bandwidth, so the M3 Ultra's 819GB/s is just a modest bump. Though the M4's additional 146GB/s is indeed a more noticeable improvement.

Also should note that 800/819GB/s of memory bandwidth is actually VERY usable for LLMs. Consider that a 4090 is just a hair above 1000GB/s

Re: Apple M3 Ultra

#513

Post by kllrnohj »

matt-p wrote: post #453
dheera wrote: post #375

Earlier quoted context omitted.

It will cost 4X what it costs to get 512GB on an x86 server motherboard.

Not really like for like. The pricing isn't as insane as you'd think, 96 to 256GB is 1500 which isn't 'cheap' but, it could be worse. All in 5,500 gets you a ultra with 256GB memory, 28 cores, 60 GPU cores, 10Gb network - I think you'd be hard pushed to build a server for less.

5,500 easily gets me either vastly more CPU cores if I care more about that or a vastly faster GPU if I care more about that. Or for both a 9950x + 5090 (assuming you can actually find one in stock) is ~$3000 for the pair + motherboard, leaving a solid $2500 for whatever amount of RAM, storage, and networking you desire.

The M3 strikes a very particular middle ground for AI of lots of RAM but a significantly slower GPU which nothing else matches, but that also isn't inherently the right balance either. And for any other workloads, it's quite expensive.

Re: Apple M3 Ultra

#514

Post by choilive »

hedora wrote: post #380
gjsman-1000 wrote: post #315

Earlier quoted context omitted.

Nah, in many businesses, everything is on a schedule. For desktop computers, a common cycle is 4 years. For servers, maybe a little longer, but not by much. After that date arrives, it’s liquidate everything and rebuild. Having things consistently work is much cheaper than down days caused by your ancient equipment. Apple’s SSDs will make it to 5 years no problem - and more likely, 10-15 years.

At my last N jobs, companies built high end server farms and carefully specced all the hardware. Then they looked at SSD specs and said “these are all fine”. Fast forward 2 years: The $50-$250K machines have a 100% drive failure rate, and some poor bastard has to fly from data center to data center to swap the $60 drive for a $120 one, then re-rack and re-image each machine. Anyway, soldering a decent SSD to the moth…

What company was specc'ing out a 6 figure machine just to put in a consumer class SSD?

Re: Apple M3 Ultra

#515

Post by aurareturn »

hbbio wrote: post #440
smith7018 wrote: post #377

Earlier quoted context omitted.

You can build an x86 machine that can fully run DeepSeek R1 with 512GB VRAM for ~$2,500?

How would you compare the tok/sec between this setup and the M3 Max?

3.5 - 4.5 tokens/s on the $2,000 AMD Epyc setup. Deepseek 671b q4.

The AMD Epyc build is severely bandwidth and compute constrained.

~40 tokens/s on M3 Ultra 512GB by my calculation.

Re: Apple M3 Ultra

#516

Post by adrian_b »

InTheArena wrote: post #100
jeffbee wrote: post #93

Earlier quoted context omitted.

That doesn't sound right. The marginal cost of +768GB of DDR5 ECC memory in an EPYC system is < $5k.

GPU accessible RAM.

In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher.

Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000.

Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.

Even for computationally-limited programs, the difference between server CPUs and consumer GPUs is not great. One Epyc CPU may have about the same number of FP32 execution units as an RTX 4070, while running at a higher clock frequency (but it lacks the tensor units of an NVIDIA GPU, which can greatly accelerate the execution, where applicable).

Re: Apple M3 Ultra

#517

Post by knowitnone »

gjsman-1000 wrote: post #263
diggan wrote: post #218

Earlier quoted context omitted.

Lack of focus on quality of software affects all types of workloads, not just consumer-oriented or professional-oriented in isolation.

Nah, if I ever wrote an article about the software crisis on the Linux desktop, there’d be flames here making Apple’s issues look small.

if you do write something, please seperate enterprise, developer, end user, embedded/RT because they all have different requirements.

Re: Apple M3 Ultra

#518

Post by PeterStuer »

cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

Is this on chip memory? From the 800GB/s I would guess more likely a 512bit bus (8 channel) to DDR5 modules. Doing it on a quad channel would just about be possible, but really be pushing the envelope. Still a nice thing.

As for practicality, which mainstream applications would benefit from this much memory paired with a nice but relative mid compute? At this price-point (14K for a full specced system), would you prefer it over e.g. a couple of NVIDIA project DIGITS (assuming that arrives on time and for around the announced the 3K price-point)?

Re: Apple M3 Ultra

#519

Post by rfoo »

ein0p wrote: post #467
valine wrote: post #408

Earlier quoted context omitted.

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

What people who did not actually work with this stuff in practice don't realize is the above statement only holds for batch size 1, sequence size 1. For processing the prompt you will need to read all the weights (which isn't a problem, because prefill is compute-bound, which, in turn is a problem on a weak machine like this Mac or an "EPYC build" someone else mentioned). Even for inference, batch size greater than 1…

For decode, MoE is nice for either bs=1 (decoding for a single user), or bs= (do EP to efficiently serve a large amount of users).

Anything in between suffers.

Re: Apple M3 Ultra

#520

Post by seanmcdirmid »

kllrnohj wrote: post #513
matt-p wrote: post #453

Earlier quoted context omitted.

Not really like for like. The pricing isn't as insane as you'd think, 96 to 256GB is 1500 which isn't 'cheap' but, it could be worse. All in 5,500 gets you a ultra with 256GB memory, 28 cores, 60 GPU cores, 10Gb network - I think you'd be hard pushed to build a server for less.

5,500 easily gets me either vastly more CPU cores if I care more about that or a vastly faster GPU if I care more about that. Or for both a 9950x + 5090 (assuming you can actually find one in stock) is ~$3000 for the pair + motherboard, leaving a solid $2500 for whatever amount of RAM, storage, and networking you desire. The M3 strikes a very particular middle ground for AI of lots of RAM but a significantly slower G…

You'll need a couple of 32GB 5090s to run a quantized 70B model, maybe 4 to run a 70b model without quantization, forget about anything larger than that. A huge model might run slow on a M3 Ultra, but at least you can run it all.

I have a Max M3 (the non-binned one), and I feel like 64GB or 96GB is within the realm of enabling LLMs that run reasonable fast on it (it is also a laptop, so I can do things on planes or trips). I thought about the Ultra, if you have 128GB for a top line M3 Ultra, the models that you could fit into memory would run fairly fast. For 512GB, you could run the bigger models, but not very quickly, so maybe not much point (at least for my use cases).

Post reply on HN