Earlier quoted context omitted.
$10k to run a 4 bit quantized model. Ouch.
[flagged]
I'm downvoting you because your use of language is so annoying, not because I work for Apple.
Apple M3 Ultra
941–950 of 1001 posts
Re: Apple M3 Ultra
#942Earlier quoted context omitted.
> There is absolutely no way that the different costs anywhere near that much at the fab. price premium probably, but chip lithography errors (thus, yields) at the huge memory density might be partially driving up the cost for huge memory.
> but chip lithography errors (thus, yields) at the huge memory density might be partially driving up the cost for huge memory. Apple's not having TSMC fab a massive die full of memory. They're buying a bunch of small dies of commodity memory and putting them in a package with a pair of large compute dies. How many of those small commodity memory dies they use has nothing to do with yield.
Is there a teardown link available for what you wrote? If so, that’s interesting.
Re: Apple M3 Ultra
#943512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
I'm not sure that unified memory is particularly relevant for that-- so e.g. on zen4/zen5 epyc there is more than enough arithmetic power that LLM inference is purely memory bandwidth limited.
On dual (SP5) Epyc I believe the memory bandwidth is somewhat greater than this apple product too... and at apple's price points you can have about twice the ram too.
Presumably the apple solution is more power efficient.
Re: Apple M3 Ultra
#944512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
"unified memory" funny that people think this is so new, when CRAY had Global Heap eons ago...
You mean the room sized super computer than sold tens of units?
Re: Apple M3 Ultra
#945Earlier quoted context omitted.
The difference between Deepseek-r1:70b (edit: actually 32b) running on an M4 Pro (48 GB unified RAM, 14 CPU cores, 20 GPU cores) and on an AMD box (64 GB DDR4, 16 core 5950X, RTX 3080 with 10 GB of RAM) is more than a factor of 2. The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing. The M4's prompt parsing took significantly lo…
You're adding detail that's not relevant to anything I said. I was saying this statement: > VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis. Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.
Sure, this is technically correct, but somewhere there's a line of practicality. Running off a CPU (especially with swap) will be past that line.
Otherwise, you don't even need a computer. Pen and paper is plenty.
For all practical purposes, VRAM is a limiting factor.
Re: Apple M3 Ultra
#946Who is this made for? Who needs a personal computer this powerful? Not trying to be funny - it's a genuine question. Gamers don't generally use a mac because of the lack of games and I'm guessing those who are really into LLMs use Linux for the flexibility. Video editing can be done on much cheaper hardware. Very rich LLM enthusiasts who wants to try out mac?
Swift developers.
Re: Apple M3 Ultra
#947512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
Agree. Finally I can have several hundred browser tabs open simultaneously with no performance degradation.
My M1 Max regularly pushes 1000+ tabs without breaking a sweat, I feel like this particular metric is no longer useful now that background tab memory is almost always unloaded by the browser.
Re: Apple M3 Ultra
#948Re: Apple M3 Ultra
#949Earlier quoted context omitted.
If you're going to overthrow your entire AI workflow to use a different API anyway, surely the AMD Instinct accelerator cards make more sense. They're expensive, but also a lot faster, and you don't need to deal with making your code work on macOS.
I don't think API has any value because writing software is free and hardware for ML is super expensive.
> writing software is free
Don't tell my boss! I still get paid.