Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

681–690 of 1001 posts

Re: Apple M3 Ultra

#681

Post by stego-tech »

hinkley wrote: post #585
stego-tech wrote: post #228

Earlier quoted context omitted.

> This hardware is really being held back by the operating system at this point. It really is. Even if they themselves won't bring back their old XServe OS variant, I'd really appreciate it if they at least partnered with a Linux or BSD (good callout, ryao) dev to bring a server OS to the hardware stack. The consumer OS, while still better (to my subjective tastes) than Windows, is increasingly hampered by bloat and…

I miss the XServe almost as much as I miss the Airport Extreme.

I feel like Apple and Ubiquiti have a missed collaboration opportunity on the latter point, especially with the latter's recent UniFi Express unit. It feels like pairing Ubiquiti's kit with Apple's Homekit could benefit both, by making it easier for Homekit users to create new VLANs specifically for Homekit devices, thereby improving security - with Apple dubbing the term, say, "Secure Device Network" or some marketingspeak to make it easier for average consumers to understand. An AppleTV unit could even act as a limited CloudKey for UniFi devices like Access Points, or UniFi Cameras to connect/integrate as Homekit Cameras.

Don't get me wrong, I wouldn't use that feature (I prefer self-hosting it all myself), but for folks like my family members, it'd be a killer addition to the lineup that makes my life supporting them much easier.

Re: Apple M3 Ultra

#682

Post by wmf »

atwrk wrote: post #531
wmf wrote: post #434

Earlier quoted context omitted.

The weights don't go over the network so performance is OK.

If I'm not mistaken, each token produced roughly equals the whole model in memory transfers (the exception being MoE models). That's why memory bandwidth is so important in the first place, or not?

My understanding is that if you can store 1/Nth of the weights in RAM on each of the N nodes then there's no need to send the weights over the network.

Re: Apple M3 Ultra

#683

Post by manmal »

choilive wrote: post #581
hereonout2 wrote: post #563

Earlier quoted context omitted.

Does it work like that though at this larger scale? 512GB of VRAM would be across multiple NVIDIA cards, so the bandwidth and access is parallelized. But here it looks more of a bottleneck from my (admittedly naive) understanding.

For inference the bandwidth is generally not parallelized because the weights need to go through the model layer by layer. The most common model splitting method is done by assigning each GPU a subset of the LLM layers and it doesn't take much bandwidth to send model weights via PCIE to the next GPU.

My understanding is that the GPU must still load its assigned layer from VRAM into registers and L2 cache for every token, because those aren’t large enough to hold a significant portion. So naively, for a 24GB layer, you‘d need to move up to 24GB for every token.

Re: Apple M3 Ultra

#684

Post by kllrnohj »

stego-tech wrote: post #661
kllrnohj wrote: post #347

Earlier quoted context omitted.

> By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP That nvidia GPU setup will actually have the compute grunt to make use of the RAM, though, which this M3 Ultra probably realistically doesn't. After all, if the only thing that mattered was RAM then the 2TB you can shove into an Epyc or Xeon would already be dominating the AI industry. But they aren't, because it…

You're forgetting what Apple's been baking into their silicon for (nearly? over?) a decade: the Neural Processing Unit (NPU), now called the "Neural Engine". That's their secret sauce that makes their kit more competitive for endpoint and edge inference than standard x86 CPUs. It's why I can get similarly satisfying performance on my old M1 Pro Macbook Pro with a scant 16GB of memory as I can on my 10900k w/ 64GB RAM…

Uh.... I must be missing something here, because you're hyping up Apple's NPU only to show it getting absolutely obliterated by the equally old 3090? Your 10900K having 64gb of RAM is also irrelevant here...

Re: Apple M3 Ultra

#685

Post by ldng »

Well, a shame for Apple, a lot of the rest of the world is going to boycott american products after such level of treacherousness.

Re: Apple M3 Ultra

#686

Post by stego-tech »

kllrnohj wrote: post #684
stego-tech wrote: post #661

Earlier quoted context omitted.

You're forgetting what Apple's been baking into their silicon for (nearly? over?) a decade: the Neural Processing Unit (NPU), now called the "Neural Engine". That's their secret sauce that makes their kit more competitive for endpoint and edge inference than standard x86 CPUs. It's why I can get similarly satisfying performance on my old M1 Pro Macbook Pro with a scant 16GB of memory as I can on my 10900k w/ 64GB RAM…

Uh.... I must be missing something here, because you're hyping up Apple's NPU only to show it getting absolutely obliterated by the equally old 3090? Your 10900K having 64gb of RAM is also irrelevant here...

You're missing the the bigger picture by getting bogged down in technical details. To an end user, the difference between thirty seconds and ninety seconds is often irrelevant for things like AI, where they expect a delay while it "thinks". When taken in that context, you're now comparing a 14" laptop running off its battery, to a desktop rig gulping down ~500W according to my UPS, for a mere 66% reduction in runtime for a single query at the expense of 5x the power draw.

Sure, the desktop machine performs better, as would a datacenter server jam-packed full of Blackwell GPUs, but that's not what's exciting about Apple's implementation. It's the efficiency of it all, being able to handle modern models on comparatively "weaker" hardware most folks would dismiss outright. That's the point I was trying to make.

Re: Apple M3 Ultra

#687

Post by ZeroTalent »

InTheArena wrote: post #55
ksec wrote: post #22

Previous model of M2 Ultra had max memory of 192GB. Or 128GB for Pro and some other M3 model, which I think is plenty for even 99.9% of professional task. They now bump it to 512GB . Along with insane price tag of $9499 for 512GB Mac Studio. I am pretty sure this is some AI Gold rush.

Every single AI shop on the planet is trying to figure out if there is enough compute or not to make this a reasonable AI path. If the answer is yes, that 10k is a absolute bargain.

No, because there is no CUDA. We have fast and cheap alternatives to NVIDIA, but they do not have CUDA. This is why NVIDIA has 90% margins on its hardware.

Re: Apple M3 Ultra

#688

Post by RachelF »

ddtaylor wrote: post #636
samstave wrote: post #622

Earlier quoted context omitted.

"unified memory" funny that people think this is so new, when CRAY had Global Heap eons ago...

Why did it take so long for us to get here?

Some possible groups of reasons: 1. Until recently RAM amount was something the end user liked to configure, so little market demand. 2. Technically, building such a large system on a chip or collection of chiplets was not possible. 3. RAM speed wasn't a bottleneck for most tasks, it was IO or CPU. LLMs changed this.

Re: Apple M3 Ultra

#689

Post by tap-snap-or-nap »

All this hardware but I don't know how to best utilize it because 1) I am not a pro, and 2) The apps are not as helpful which can make complex jobs easier, which is what old apple used to do really well.

Re: Apple M3 Ultra

#690

Post by winocm »

PaulHoule wrote: post #376
hedora wrote: post #360

Earlier quoted context omitted.

In fairness, the sun and dec boxes I used back then (up to about 1999) could hold their own against intel machines. Then, one day, we built a 5 machine amd athlon xp linux cluster for $2000 ($400/machine) that beat all the unix and windows server hardware by at least 10x on $/perf. It’s nice that we have more than one viable cpu vendor these days, though it seems like there’s only one viable fab company.

In 1998-1999 I had a DEC Alpha on my desktop that was really impressive, it was a 64-bit machine a few years before you could get a 64-bit Athlon.

The Alpha architecture was 64-bit from the very beginning (though the amount of addressable virtual memory and physical memory depends on the processor implementation).

I think it goes something like:

  - 2106x/EV4: 34-bit physical, 43-bit virtual
  - 21164/EV5: 40-bit physical, 43-bit virtual
  - 21264/EV6: 44-bit physical, 48-bit virtual
The EV6 is a bit quirky as it is 43-bit by default, but can use 48-bits when I_CTL or VA_CTL is set. (the distinction of the registers is for each access type, i.e: instruction fetch versus data load/store)

The 21364/EV7 likely has the same characteristics as EV6, but the hardware reference manual seems to have been lost to time...

Post reply on HN