Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

881–890 of 1001 posts

Re: Apple M3 Ultra

#881

Post by rtwld »

jjuliano wrote: post #870

Currently, Docker does not support Metal GPUs. When running LLMs on Docker with an Apple M3 or M4 chip, they will operate in CPU mode regardless of the chip's class, as Docker only supports Nvidia and Radeon GPUs. If you're developing LLMs on Docker, consider getting a Framework laptop with an Nvidia or Radeon GPU instead. Source: I develop an AI agent framework that runs LLMs inside Docker on an M3 Max ( https://kde…

Podman does support GLU acceleration through libkrun with virtio-gpu (venus) on Mac: https://podman-desktop.io/docs/podman/gpu

Re: Apple M3 Ultra

#882

Post by kalleboo »

VVilhelmsen wrote: post #846

Who is this made for? Who needs a personal computer this powerful? Not trying to be funny - it's a genuine question. Gamers don't generally use a mac because of the lack of games and I'm guessing those who are really into LLMs use Linux for the flexibility. Video editing can be done on much cheaper hardware. Very rich LLM enthusiasts who wants to try out mac?

Modern video editing includes tasks such as AI upscaling and subtitle generation that can use lots of power

Re: Apple M3 Ultra

#883

Post by DrBenCarson »

behnamoh wrote: post #736
DrBenCarson wrote: post #656

Earlier quoted context omitted.

Apple are working on the hard problems of making AI useful (call them “agents”), not AGI 1. Small models running locally with well-established tool interfaces (“app intents”) 2. Large models running in a bespoke cloud that can securely and quickly load all relevant tokens from a device before running inference No AI lab is even close to what Apple is trying to deliver in the next ~12 months

if that were the case, then it would definitely help Apple Intelligence if the iPhone and Mac had higher amounts of RAM, but the base MacBook Pro announced by Apple a while ago had 8 GB of RAM and even the pro versions of the iPhone have 8GB whereas 12, 16, or even higher RAM is very common in android devices which helps users run relatively large language models on their devices

All iPhone SoCs made after ChatGPT launched start with 8gb, and all Mac SoCs start with 16hlgb

Re: Apple M3 Ultra

#885

Post by DrBenCarson »

827a wrote: post #796
DrBenCarson wrote: post #651

Earlier quoted context omitted.

And how much VRAM will Project Digits have?

128gb each, so two would have 256gb. Its half that of a max spec Mac Studio, but also half the price and eight times faster memory speed. Realistically which open source LLMs does 512gb over 256gb of memory unlock? My understanding is that the true bleeding edge ones like R1 won't even handle 512gb well, especially with the anemic memory speed.

I agree project digits looks to be the better all-around option for AI researchers, but I still think the Mac is better for people building products with AI

Re memory speed, digits will be at 273GB/s while the Mac Studio is at 819GB/s

Not to mention the Mac has 6 120GB/s thunderbolt 5 ports and can easily be used for video editing, app development, etc.

Re: Apple M3 Ultra

#886

Post by enether »

Can anyone with older Mac Studios/Minis comment - do you also notice a "throttling" of the hardware?

I'm not sure if this is me not maintaining it properly (e.g fans having dust block them) - but I've always got this sense that Apple throttles their older devices in some indirect ways. I experience it the most with iPhones - my old iPhone is pretty slow doing basic things despite nothing really changing on it (just the OS updating?)

So my only concern with this is - how many years until it's slow enough to annoy you into buying a new one?

Re: Apple M3 Ultra

#887

Post by egorfine »

mannyv wrote: post #547
egorfine wrote: post #496

Earlier quoted context omitted.

> why do you want this in MacOS? I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive. I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side. My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact…

Funny, I ran a bunch of Mac minis in colo for over a decade with no problems. Maybe you have a config problem? Of course, I had a LOM/KVM and redundant networking etc. They were substantially more reliable than the Dell equipment that I used in my day job for sure.

Hardware-wise I have exactly zero complaints.

Software-wise it's much different to an expected behavior. For example, macOS won't let you in over SSH until you log in via Remote Desktop. You'll get "connection closed" immediately.

Or sometimes it will.

And that depends not on the count of connection attempts or anything you can do locally but rather on the boot process somehow. Sometimes it boots in a way that permits ssh, sometimes not. The same computer, the same OS.

Then after you login on screen sharing and log out, macOS will let you in over ssh. For a few days. And then again will force you to login via GUI. Or maybe not. I have no idea what makes it.

I have trouble reading macOS logs or understanding it. It spews a few log messages per second even idle. If you grep ssh these messages contain zero actionable data, like "unsuccessful attempt" or similar.

Another complaint is that launchd reports the same "I/O error" on absolutely all error situations, from syntax error in plist to corrupt binary. Makes development and debugging of launchagents very fun.

Re: Apple M3 Ultra

#888

Post by egorfine »

duped wrote: post #561
egorfine wrote: post #496

Earlier quoted context omitted.

> why do you want this in MacOS? I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive. I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side. My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact…

What would a containerization environment on MacOS give you that you don't already have? Like concretely - what does containerization mean in the context of a MacOS user space? In Linux, it means something very specific: a user/mount/pid/network namespace, overlayfs to provide a rootfs, chroot to pivot to the new root to do your work, and port forwarding between the host/guest systems. On MacOS I don't know what cont…

On macOS probably I'd like chroot and pid/mount namespaces. I'd like to install OS and dependencies in a container and run my application from there so that it does not interfere with host OS. My app is GPU heavy and has lots of dependencies (OpenCV, LAPACK, armadillo, lots and lots) and I'd like to not pollute the host OS with it.

Also I want to run the latest OS with all security patches on the host while having a stable and known macOS version in a container given how developer-hostile Apple is.

Re: Apple M3 Ultra

#889

Post by brookst »

lynguist wrote: post #718
lhl wrote: post #560

Earlier quoted context omitted.

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

Design work on the Ultra would have started 2-3 years ago, and specs for memory at least 18 months ago. I’m not sure they had that kind of inside knowledge for what Deepseek specifically was doing that far in advance. Did Deepseek even know that long ago?

Re: Apple M3 Ultra

#890

Post by Sharlin »

tobyhinloopen wrote: post #869
Sharlin wrote: post #329

Earlier quoted context omitted.

It enables the use of giant AI models on a personal computer. Might not run too fast though. But at least it's possible at all .

What is stopping us from running these models on a PC with 512GB RAM?

You have a point; technically they aren't impossible to run if you have enough system RAM (or hell, SSD/HDD space for that mater). But in practice neither running on the CPU, nor on the GPU by constantly paging data in and out of VRAM, is a very attractive option (~10x slowdown at least).
Post reply on HN