Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

561–570 of 1001 posts

Re: Apple M3 Ultra

#561

Post by duped »

egorfine wrote: post #496
duped wrote: post #258

Earlier quoted context omitted.

> No native docker support Honest question: why do you want this in MacOS? Do you understand what docker does? (it's fundamentally a linux technology, unless you are asking for user namespaces and chroot w/o SIP on MacOS, but that doesn't make sense since the app sandbox exists). MacOS doesn't have the fundamental ecosystem problems that beget the need for docker. If the answer is "I want to run docker containers bec…

> why do you want this in MacOS? I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive. I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side. My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact…

What would a containerization environment on MacOS give you that you don't already have? Like concretely - what does containerization mean in the context of a MacOS user space?

In Linux, it means something very specific: a user/mount/pid/network namespace, overlayfs to provide a rootfs, chroot to pivot to the new root to do your work, and port forwarding between the host/guest systems.

On MacOS I don't know what containerization means short of virtualization. But you have virtualization on MacOS already, so why not use that?

Re: Apple M3 Ultra

#562

Post by aurareturn »

sgt wrote: post #543
aurareturn wrote: post #515

Earlier quoted context omitted.

3.5 - 4.5 tokens/s on the $2,000 AMD Epyc setup. Deepseek 671b q4. The AMD Epyc build is severely bandwidth and compute constrained. ~40 tokens/s on M3 Ultra 512GB by my calculation.

What kind of Nvidia-based rig would one need to achieve 40 tokens/sec on Deepseek 671b? And how much would it cost?

Around 5x Nvidia A100 80GB can fit 671b Q4. $50k just for the GPUs and likely much more when including cooling, power, motherboard, CPU, system RAM, etc.

Re: Apple M3 Ultra

#563

Post by hereonout2 »

choilive wrote: post #512
okanesen wrote: post #494

Earlier quoted context omitted.

Not that dramatic of an increase actually - the M2 Max already had 400GB/s and M2 Ultra 800GB/s memory bandwidth, so the M3 Ultra's 819GB/s is just a modest bump. Though the M4's additional 146GB/s is indeed a more noticeable improvement.

Also should note that 800/819GB/s of memory bandwidth is actually VERY usable for LLMs. Consider that a 4090 is just a hair above 1000GB/s

Does it work like that though at this larger scale? 512GB of VRAM would be across multiple NVIDIA cards, so the bandwidth and access is parallelized.

But here it looks more of a bottleneck from my (admittedly naive) understanding.

Re: Apple M3 Ultra

#564

Post by alok-g »

Two questions for the fellow HNers:

1. What are various average joe (as opposed to researchers, etc.) use cases for running powerful AI models locally vs. just using cloud AI. Privacy of course is a benefit, but it by itself may not justify upgrades for an average user. Or are we expecting that new innovation will lead to much more proliferation of AI and use cases that will make running locally more feasible?

2. With the amount of memory used jumping up, would there be a significant growth for companies making memories? If so, which ones would be the best positioned?

Thanks.

Re: Apple M3 Ultra

#565

Post by bigyabai »

fintechie wrote: post #302

IMO this is a bigger blow to the AI big boys than Deepseek's release. This is massive for local inference. Exciting times ahead for open source AI.

I don't think you understand what the "AI big boys" are in the market for.

Re: Apple M3 Ultra

#566

Post by choilive »

pavlov wrote: post #268
bigyabai wrote: post #246

Earlier quoted context omitted.

It's the Ultra chip, the same one that goes into the rackmount Mac Pro. I don't think there's much confusion as to who this is for. > there’s no sign that they’re interested in updating that for the AI age. https://security.apple.com/blog/private-cloud-compute/

I genuinely forgot the Mac Pro still exists. It’s been so long since I even saw one. And I’ve had every previous Mac tower design since 1999: G4, G5, the excellent dual Xeon, the horrible black trash can… But Apple Silicon delivers so much punch in the Studio form factor, the old school Pro has become very niche. Edit - looks like the new M3 Ultra is only available in Mac Studio anyway? So the existence of the Pro is…

never understood the hate on the trash can. Isn't the mac studio basically the same idea as the trash can but even less upgradeable?

Re: Apple M3 Ultra

#567

Post by KingOfCoders »

dheera wrote: post #450
LeifCarrotson wrote: post #446

Earlier quoted context omitted.

Yep, it's apples to oranges. But sometimes you want apples, and sometimes you want oranges, so it's all good! There's a wide spectrum of potential requirements between memory capacity, memory bandwidth, compute speed, compute complexity, and compute parallelism. In the past, a few GB was adequate for tasks that we assigned to the GPU, you had enough storage bandwidth to load the relevant scene into memory and generat…

Sure, if you want to do training get an NVIDIA card. My point is that it's not worth comparing either Mac or CPU x86 setup to anything with NVIDIA in it. For inference setups, my point is that instead of paying $10000-$15000 for this Mac you could build an x86 system for The "+$4000" for 512GB on the Apple configurator would be "+$1000" outside the Apple world.

But this is how it wonderfully works. +$4000 does two things: 1. Make Apple very very rich 2. Make people think this is better than a $10k EPYC. Win-Win for Apple. At the point when you have convinced that you are the best, higher price just means people think you are even better.

Re: Apple M3 Ultra

#568

Post by fauigerzigerk »

hedora wrote: post #437
p_ing wrote: post #250

Earlier quoted context omitted.

No one would want OOTB Python support. You'd be stuck on a version you didn't want to use.

I want it. That way, like code I write in any other language, it’ll run reliably on other people’s machines a few years from now. I avoid writing python, so I’m usually the “other people” in that sentence.

>it’ll run reliably on other people’s machines a few years from now

That's optimistic. What if the system Python gets upgraded? For some reason, Python libraries tend to be super picky about the Python versions they support (not just Python 2 vs 3).

Re: Apple M3 Ultra

#569

Post by epolanski »

Can anybody ELI5 why aren't there multi gpu builds to run LLMs locally?

It feels like one should be able to build a good machine for 3/4k if not less with 6 16GB mid level gaming GPUs.

Re: Apple M3 Ultra

#570

Post by bob1029 »

FloatArtifact wrote: post #301
cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…

> The question is if a llm will run with usable performance at that scale?

For the self-attention mechanism, memory bandwidth requirements scale ~quadratically with the sequence length.

Post reply on HN