Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

531–540 of 1001 posts

Re: Apple M3 Ultra

#531

Post by atwrk »

wmf wrote: post #434
atwrk wrote: post #355

Earlier quoted context omitted.

But 80Gbit/s is way slower than even regular dual channel RAM, or am I missing something here? That would mean the LLM would be excruciatingly slow. You could get an old EPYC for a fraction of that price and have more performance.

The weights don't go over the network so performance is OK.

If I'm not mistaken, each token produced roughly equals the whole model in memory transfers (the exception being MoE models). That's why memory bandwidth is so important in the first place, or not?

Re: Apple M3 Ultra

#532

Post by valine »

bick_nyers wrote: post #458
valine wrote: post #408

Earlier quoted context omitted.

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

Just to add onto this point, you expect different experts to be activated for every token, so not having all of the weights in fast memory can still be quite slow as you need to load/unload memory every token.

Probably better to be moving things from fast memory to faster memory than from slow disk to fast memory.

Re: Apple M3 Ultra

#533

Post by egorfine »

awestroke wrote: post #418
screye wrote: post #407

How does the 500gb vram compare with 8xA100s ? ($15/hr rentals) If it is equivalent, then the machine pays for itself in 300 hours. That's incredible value.

A100 has 10x or so higher mem bandwidth

Per nvidia [1] A100 has memory bandwidth up to 2,039. So not 10x, more like 2x.

[1] https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...

Re: Apple M3 Ultra

#534

Post by varispeed »

A4ET8a8uTh0_v2 wrote: post #62
varispeed wrote: post #34

Earlier quoted context omitted.

You know that memory can be "easily" de-soldered and soldered at home? The issue is availability of chips and most likely you have to know which components to change so the new memory is recognised. For instance that could be changing a resistor to different value or bridging certain pads.

This viewpoint is interesting. It is not exactly inaccurate, but it does appear to be missing a point. Soldering in itself is a valuable and useful skill, but I can't say you can just get in and start de-soldering willy-nilly as opposed to opening a box and upgrading ram by plopping stuff in a designated spot. What if both are an issue?

Do you know that "plopping stuff in a designated spot" can also be out of reach to some people? I know plenty who would give their computer to a tech do to the upgrade for them even if they are shown in person how to do all the steps. Soldering is just one step (albeit fairly big) above that. But the fact this can be done at home with fairly inexpensive tools, means tech person with reasonable skill could do it, so such upgrade could be accessible in computer/phone repair shop if parts were available to do so. Soldering is not a barrier - what I am trying to say.

Re: Apple M3 Ultra

#536

Post by naikrovek »

devmor wrote: post #262
naikrovek wrote: post #253

Earlier quoted context omitted.

surprisingly, Windows containers on Windows are not run in a VM. Well, not necessarily; they can be. It is definitely odd that Macs have no native container support, though, especially when you learn that Windows does.

That is an important point, I didn't really think of it since I've never had a reason to use Windows containers.

that's ok, no one thinks of windows, and fewer people than that would ever use a windows container.

Re: Apple M3 Ultra

#537

Post by kstrauser »

tomn wrote: post #454
pmarreck wrote: post #200

Earlier quoted context omitted.

> lack of robust python support There is no such thing. Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers would you use? And how would you prevent "project collision" (where one Python project bumps into another one and one just stops working)? Example: SSL library differences across projects is a notorious culprit. Python is garbage and I don't understand w…

This is incoherent to me. Your complaints are about packaging, but the elixir wrapper doesn't deal with that in any way -- it just wraps UV, which you could use without elixir. What am I missing? Also, typically when people say things like > Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers It means they have been trapped in a cycle of thinking "just one mor…

I agree. Python certainly had its speedbumps, but it's utterly manageable today and has been for years and years. It seems like people get hung up on there not being 1 official way to do things, but I think that's been great, too: the competition gave us nice things like Poetry and UV. The odds are slim that a Rust tool would've been accepted as the official Python.org-supplied system, but now we have it.

There are reasons to want something more featureful than plain pip. Even without them, pip+virtualenv has been completely usable for, what, 15 years now?

Re: Apple M3 Ultra

#538

Post by rbanffy »

exabrial wrote: post #203
InTheArena wrote: post #29

Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

> The CPUs have zero competition in terms of speed, memory bandwidth.

Maybe not at the same power consumption, but I'm sure mid-range Xeons and EPYCs mop the floor with the M3 Ultra in CPU performance. What the M3 Ultra has that nobody else comes close is a decent GPU near a pool of half a terabyte of RAM.

Re: Apple M3 Ultra

#539

Post by Der_Einzige »

ein0p wrote: post #467
valine wrote: post #408

Earlier quoted context omitted.

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

What people who did not actually work with this stuff in practice don't realize is the above statement only holds for batch size 1, sequence size 1. For processing the prompt you will need to read all the weights (which isn't a problem, because prefill is compute-bound, which, in turn is a problem on a weak machine like this Mac or an "EPYC build" someone else mentioned). Even for inference, batch size greater than 1…

No one who is using this for home use cares about anything except batch size 1 sequence size 1.

Re: Apple M3 Ultra

#540

Post by alberth »

NorwegianDude wrote: post #217
InTheArena wrote: post #55

Earlier quoted context omitted.

Every single AI shop on the planet is trying to figure out if there is enough compute or not to make this a reasonable AI path. If the answer is yes, that 10k is a absolute bargain.

Not much to figure out. It's 2x M4 Max, so you need 100 of these to match the TOPS of even a single consumer card like the RTX 5090.

> It's 2x M4 Max

Not exactly though.

This can have 512GB unified memory, 2x M4 Max can only have 128GB total (64GB each).

Post reply on HN