Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

541–550 of 1001 posts

Re: Apple M3 Ultra

#541

Post by dragonwriter »

valine wrote: post #535
bick_nyers wrote: post #461

Earlier quoted context omitted.

About $12k when Project Digits comes out.

That will only have 128GB of unified memory

128GB for 3K; per the announcement their ConnectX networking allows two Project Digits devices to be plugged into eachother and work together as one device giving you 256GB for $6k, and, AFAIK, existing frameworks can split models across devices, as well, hence, presumably, the upthread suggestion that Project Digits would provide 512GB for $12k, though arguably the last step is cheating.

Re: Apple M3 Ultra

#542

Post by aurareturn »

agloe_dreams wrote: post #206
lauritz wrote: post #46

They update the Studio to M3 Ultra now, so M4 Ultra can presumably go directly into the Mac Pro at WWDC? Interesting timing. Maybe they'll change the form factor of the Mac Pro, too? Additionally, I would assume this is a very low-volume product, so it being on N3B isn't a dealbreaker. At the same time, these chips must be very expensive to make, so tying them with luxury-priced RAM makes some kind of sense.

My understanding was that Apple wanted to figure out how to build systems with multi-SOCs to replace the Ultra chips. The way it is currently done means that the Max chips need to be designed around the interconnect. Theoretically speaking, a multi-SOC setup could also scale beyond two chips and into a wider set of products.

I'm not sure if multi-SoC is possible because having 2 GPUs together such that the OS sees it as one big GPU is not very possible if the SoCs are separated.

Re: Apple M3 Ultra

#543

Post by sgt »

aurareturn wrote: post #515
hbbio wrote: post #440

Earlier quoted context omitted.

How would you compare the tok/sec between this setup and the M3 Max?

3.5 - 4.5 tokens/s on the $2,000 AMD Epyc setup. Deepseek 671b q4. The AMD Epyc build is severely bandwidth and compute constrained. ~40 tokens/s on M3 Ultra 512GB by my calculation.

What kind of Nvidia-based rig would one need to achieve 40 tokens/sec on Deepseek 671b? And how much would it cost?

Re: Apple M3 Ultra

#544

Post by kstrauser »

hedora wrote: post #429
kstrauser wrote: post #296

Earlier quoted context omitted.

Virtualenv’s been a thing for many years, it’s built into Python, and it solves all that without adding additional tooling. And if you’re genuinely asking, everything’s converging toward uv. If you pick only one, use that and be done with it.

I’ve been using virtualenv for a decade, and we use uv at work. Neither fixed anything. They just make it slightly less painful to deal with python scripts’ constant bitrot. They also make python uniquely difficult to dockerize.

That's so completely, diametrically opposite of my experience with both that I can't help but wonder how it ended up there.

> They also make python uniquely difficult to dockerize.

  RUN pip install uv && uv sync
Tada, done. No, seriously. That's the whole invocation.

Re: Apple M3 Ultra

#545

Post by zitterbewegung »

PeterStuer wrote: post #518
cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

Is this on chip memory? From the 800GB/s I would guess more likely a 512bit bus (8 channel) to DDR5 modules. Doing it on a quad channel would just about be possible, but really be pushing the envelope. Still a nice thing. As for practicality, which mainstream applications would benefit from this much memory paired with a nice but relative mid compute? At this price-point (14K for a full specced system), would you pre…

NVIDIA project DIGITS has 128 GB LPDDR5x coherent unified system memory at a 273 Gb/s memory bus speed.

Re: Apple M3 Ultra

#546

Post by aurareturn »

adrian_b wrote: post #516
InTheArena wrote: post #100

Earlier quoted context omitted.

GPU accessible RAM.

In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher. Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000. Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple s…

  Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.
Source on this? CPUs would be very compute constrained.

Re: Apple M3 Ultra

#547

Post by mannyv »

egorfine wrote: post #496
duped wrote: post #258

Earlier quoted context omitted.

> No native docker support Honest question: why do you want this in MacOS? Do you understand what docker does? (it's fundamentally a linux technology, unless you are asking for user namespaces and chroot w/o SIP on MacOS, but that doesn't make sense since the app sandbox exists). MacOS doesn't have the fundamental ecosystem problems that beget the need for docker. If the answer is "I want to run docker containers bec…

> why do you want this in MacOS? I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive. I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side. My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact…

Funny, I ran a bunch of Mac minis in colo for over a decade with no problems. Maybe you have a config problem?

Of course, I had a LOM/KVM and redundant networking etc. They were substantially more reliable than the Dell equipment that I used in my day job for sure.

Re: Apple M3 Ultra

#550

Post by lostmsu »

bastardoperator wrote: post #479
radlad wrote: post #422

Earlier quoted context omitted.

It is much slower than nVidia, but for a lot of personal-use LLM scenarios, it's very workable. And it doesn't need to be anywhere near as fast considering it's really the only viable (affordable) option for private, local inference, besides building a server like this, which is no faster: https://news.ycombinator.com/item?id=42897205

It's fast enough for me to cancel monthly AI services on a mac mini m4 max.

Hm, the AI services over 5 years cost half of m4 max minimal configuration which can barely run severely lobotomized LLaMA 70B. And they provide significantly better models.
Post reply on HN