Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

331–340 of 1001 posts

Re: Apple M3 Ultra

#331

Post by llm_nerd »

dagmx wrote: post #293
llm_nerd wrote: post #195

Earlier quoted context omitted.

Using the NPU numbers grossly overstates the AI performance of the Apple Silicon hardware, so they're actually giving Apple the benefit of the doubt. Most AI training and inference (including generative AI) is bound by large scale matrix MACs. That's why nvidia fills their devices with enormous numbers of tensor cores and Apple / Qualcomm et al are adding NPUs, filling largely the same gap. Only nvidia's not only are…

Care to share the TOPs numbers for the Apple GPUs and show how this would “grossly overstate” the numbers? Apple won’t compete with NVIDIA, I’m not arguing that. But your opening line will only make sense if you can back up the numbers and the GPU performance is lower than the ANE TOPS.

Tensor / neural cores are very easy to benchmark and give a precise number because they do a single well-defined thing at a large scale. So GPU numbers are less common and much more use-specific.

However the M2 Ultra GPU is estimated, with every bit of compute power working together, at about 26 TOPS.

Re: Apple M3 Ultra

#332

Post by nick_ »

hedora wrote: post #287
exabrial wrote: post #203

Earlier quoted context omitted.

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

The last I checked, AMD was outperforming Apple perf/dollar on the high end, though they were close on perf/watt for the TDPs where their parts overlapped. I’d be curious to know if this changes that. It’d take a lot more than doubling cores to take out the very high power AMD parts, but this might squeeze them a bit. Interestingly, AMD has also been investing heavily in unified RAM. I wonder if they have / plan an S…

Same. I'm not sure what to make of the various claims. I personally defer to this table in general: https://www.cpubenchmark.net/power_performance.html.

I'm not sure how those benchmarks translate to common real world use cases.

Re: Apple M3 Ultra

#333

Post by galad87 »

hoppp wrote: post #288
exabrial wrote: post #203

Earlier quoted context omitted.

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

What about serviceability? These come with soldered in ssd? That would be an issue for server use, Its too expensive to throw it away all for a broken ssd.

No, the SSD isn't soldered, it has got one or two removable modules: https://everymac.com/systems/apple/mac-studio/mac-studio-faq...

Re: Apple M3 Ultra

#335

Post by eigenspace »

jsheard wrote: post #89
eigenspace wrote: post #72

Earlier quoted context omitted.

Framework said that when they built a Strix Halo machine, AMD assigned an engineer to work with them on seeing if there's a way to get CAMM2 memory working with it, and after a bunch of back and forth it was decided that CAMM2 still made the traces too long to maintain proper signal integrity due to the 256 bit interface. These machines have a 512 bit interface, so presumably even worse.

Yeah, but AMDs memory controllers are really finnicky. That might have been more of a Strix Halo issue than a CAMM2 issue.

Entirely possible. Obviously Apple wouldn't have been interested in letting you upgrade the RAM even if it was doable.

I'd love to have more points of comparison available, but Strix Halo is the most analogous chip to an M-series chip on the market right now from a memory point of view, so it's hard to really know anything.

I very much hope CAMM2 or something else can be made to work with a Strix-like setup in the future, but I have my doubts.

Re: Apple M3 Ultra

#336

Post by kjreact »

numpad0 wrote: post #191
InTheArena wrote: post #100

Earlier quoted context omitted.

GPU accessible RAM.

moot point if tok/s benchmark results are the same or worse.

Are the benchmarks worse? Running LLMs in system memory is rather painful. I am having a hard time finding benchmarks for running large models using system memory. Can you point me to some benchmarks you’re referring to?

Re: Apple M3 Ultra

#337

Post by dagmx »

llm_nerd wrote: post #331
dagmx wrote: post #293

Earlier quoted context omitted.

Care to share the TOPs numbers for the Apple GPUs and show how this would “grossly overstate” the numbers? Apple won’t compete with NVIDIA, I’m not arguing that. But your opening line will only make sense if you can back up the numbers and the GPU performance is lower than the ANE TOPS.

Tensor / neural cores are very easy to benchmark and give a precise number because they do a single well-defined thing at a large scale. So GPU numbers are less common and much more use-specific. However the M2 Ultra GPU is estimated, with every bit of compute power working together, at about 26 TOPS.

Could you provide a link for that TOPS count? (And specifically TOPs with comparable unit sizes since NVIDIA and Apple did not use the same units till recently)

The only similar number I can find is for TFLOPS vs TOPS

Again I’m not saying the GPU will be comparable to an NVIDIA one, but that the comparison point isn’t sensible in the comments I originally replied to.

Re: Apple M3 Ultra

#338

Post by gatienboquet »

whimsicalism wrote: post #278
gatienboquet wrote: post #78

Earlier quoted context omitted.

LLMs are primarily "memory-bound" rather than "compute-bound" during normal use. The model weights (billions of parameters) must be loaded into memory before you can use them. Think of it like this: Even with a very fast chef (powerful CPU/GPU), if your kitchen counter (VRAM) is too small to lay out all the ingredients, cooking becomes inefficient or impossible. Processing power still matters for speed once everythin…

Transformers are typically memory- bandwidth bound during decoding. This chip is going to have a much worse memory b/w than the nvidia chips. My guess is that these chips could be compute-bound though given how little compute capacity they have.

VRAM capacity is the initial gatekeeper, then bandwidth becomes the limiting factor.

Re: Apple M3 Ultra

#339

Post by cubefox »

gield wrote: post #198
_alex_ wrote: post #152

apple keeps talking about the Neural Engine. Does anything actually use it? Seems like all the current LLM and Stable Diffusion packages (including MLX) use the GPU.

Face ID, taking pictures, Siri, ARKit, voice-to-text transcription, face recognition and OCR in photos, noise filtering, ...

These have been possible in much smaller smartphone chips for years.

Re: Apple M3 Ultra

#340

Post by officeplant »

dwedge wrote: post #146
darthrupert wrote: post #110

Earlier quoted context omitted.

Yeah, if only Apple at least semi-supported Linux, their computers would have no competition.

I've been buying and using MBP for 6 or 7 years now, and just assumed I could run Linux on one if I wanted to. I just spent a couple of days trying to get a 2018 MBP working with Linux and found out [edit to clarify] that my other ARM MBP basically won't work. I just want a break from MacOS, I'll be buying a Thinkpad and will probably never come back. This isn't my moaning, I understand it's their market, but if thei…

Loved my M1 mini, loved my M2 air. I've moved on to 2024 HP Elitebook with an AMD R7 8840U, 1TB replaceable NVME, 32gb of socketed DDR5. 14in laptop with a serviceable enough 1920x1200 matte screen. $800 and a 3 hour drive to the nearest Microcenter. I gave Apple another try (refused apple from 2009-2020 because of the nvidia era issues) and I just can't stomach living off of piles of external drives anymore to make up for their lack luster storage space on the affordable units.

The HP Elitebook was on Ubuntu's list of compatible tested laptops and came in hundreds of dollars less than a Thinkpad. Most of the comparably priced on sale T14's I could find were all crap Intel spec'd ones.

Months in I don't regret it at all and Linux support has been fantastic even for a fairly newer Ryzen chip and not the latest kernel. (I stick to LTS releases of most Distros) Shoving in 4TB of NVME storage and 96GB of DDR5 should I feel the need to upgrade would still put me only around $1300 invested in this machine.

Post reply on HN