Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

921–930 of 1001 posts

Re: Apple M3 Ultra

#921

Post by pmarreck »

kstrauser wrote: post #544
hedora wrote: post #429

Earlier quoted context omitted.

I’ve been using virtualenv for a decade, and we use uv at work. Neither fixed anything. They just make it slightly less painful to deal with python scripts’ constant bitrot. They also make python uniquely difficult to dockerize.

That's so completely, diametrically opposite of my experience with both that I can't help but wonder how it ended up there. > They also make python uniquely difficult to dockerize. RUN pip install uv && uv sync Tada, done. No, seriously. That's the whole invocation.

So you've never encountered a stale python project that used to work and had difficulties getting it up and running again?

Do you actually try new Python projects out with git-clone, or do you just use the same 3 Python projects for years at a time (all regularly)?

That might explain the difference in experiences

Re: Apple M3 Ultra

#922

Post by pmarreck »

whimsicalism wrote: post #309
pmarreck wrote: post #200

Earlier quoted context omitted.

> lack of robust python support There is no such thing. Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers would you use? And how would you prevent "project collision" (where one Python project bumps into another one and one just stops working)? Example: SSL library differences across projects is a notorious culprit. Python is garbage and I don't understand w…

these are solved problems now, check back in. uv is now the standard

Cool. To paraphrase an old XKCD (https://xkcd.com/927/), now there are 16 standards!

Re: Apple M3 Ultra

#923

Post by tgma »

saagarjha wrote: post #833
tgma wrote: post #821

Earlier quoted context omitted.

Pretty sure this has absolutely nothing to do with Deepseek and even local LLM at large, which has been a thing for a while and an obvious use case original Llama leak and llama.cpp coming around. Fact is Mac Pros in the Intel days supported 1.5TB RAM in some configurations[1] and that was 6 years ago expectations of their high end customer base. They needed to address the gap for those customers so they would have s…

The thing that people are excited about here is unified memory that the GPU can address. Mac Pro had discrete GPUs with their own memory.

I understand why they are excited about it—just pointing out it is a happy coincidence. They would have and should have made such a product to address the need of RAM users alone, not VRAM in particular, before they have a credible case to cut macOS releases on Intel.

Re: Apple M3 Ultra

#924

Post by ngneer »

astrange wrote: post #762
exabrial wrote: post #203

Earlier quoted context omitted.

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

The interesting difference between x86 and ARM is security, not performance, btw.

Hardly. x86 and ARM follow similar security underpinnings and outcomes. ARM has TrustZone, x86 has TEEs. I cannot think of a single attack demonstrated on x86 and not on ARM or vice versa. Could you please cite one?

Re: Apple M3 Ultra

#925

Post by a1o »

j45 wrote: post #781
a1o wrote: post #739

Earlier quoted context omitted.

Any ideas on power consumption? I wonder how much power would that use. It looks like it would be more efficient than everything else that currently exists.

Looks like up to 480W listed here https://www.apple.com/mac-studio/specs/

Thanks!!

Re: Apple M3 Ultra

#926

Post by happyopossum »

lynguist wrote: post #718
lhl wrote: post #560

Earlier quoted context omitted.

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

> they specifically built this M3 Ultra for DeepSeek R1 4-bit

Which came out in what, mid January? Yeah, there's no chance Apple (or anyone) has built a new chip in the last 45 days.

Re: Apple M3 Ultra

#927

Post by unsatchmo »

wmf wrote: post #682
atwrk wrote: post #531

Earlier quoted context omitted.

If I'm not mistaken, each token produced roughly equals the whole model in memory transfers (the exception being MoE models). That's why memory bandwidth is so important in the first place, or not?

My understanding is that if you can store 1/Nth of the weights in RAM on each of the N nodes then there's no need to send the weights over the network.

You're correct about the weights: each machine could in fact store all of the weights. However I think you still have to transfer the activations and the KV-Cache while performing inference.

Re: Apple M3 Ultra

#928

Post by bastardoperator »

Matl wrote: post #610
staticman2 wrote: post #511

Earlier quoted context omitted.

Smaller, dumber models are faster than bigger, slower ones. What model do you find fast enough and smart enough?

Not OP but I am finding the Qwen 2.5 32b distilled with DeepSeek R1 model to be a good speed/smartness ratio on the M4 Pro Mac Mini.

I'm running the same exact models.

Re: Apple M3 Ultra

#929

Post by tobyhinloopen »

Sharlin wrote: post #890
tobyhinloopen wrote: post #869

Earlier quoted context omitted.

What is stopping us from running these models on a PC with 512GB RAM?

You have a point; technically they aren't impossible to run if you have enough system RAM (or hell, SSD/HDD space for that mater). But in practice neither running on the CPU, nor on the GPU by constantly paging data in and out of VRAM, is a very attractive option (~10x slowdown at least).

So the only reason the mac is faster is because the RAM is accessible by its GPU, right? Not because the RAM is faster than regular RAM, because AFAIK it isn't far off from workstation RAM speeds.

Re: Apple M3 Ultra

#930

Post by rlt »

cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

Is putting RAM on the same chip as processing economical?

I would have assumed you’d want to save the best process/node for processing, and could use a less expensive processes for RAM.

Post reply on HN