Earlier quoted context omitted.
It will cost 4X what it costs to get 512GB on an x86 server motherboard.
What would it cost to get 512GB of VRAM on an Nvidia card? That’s the real comparison.
About $12k when Project Digits comes out.
Apple M3 Ultra
461–470 of 1001 posts
Re: Apple M3 Ultra
#462Earlier quoted context omitted.
And how is it only £9,699.00!! Does that dollar price include sales tax or are Brits finally getting a bargain?
The US prices never include state sales tax IIRC. Maybe we're finally getting some parity.
You could always buy one at an apple store without sales tax (e.g. Portland Oregon). But they might not have that one in stock...
Re: Apple M3 Ultra
#463Earlier quoted context omitted.
Nah, in many businesses, everything is on a schedule. For desktop computers, a common cycle is 4 years. For servers, maybe a little longer, but not by much. After that date arrives, it’s liquidate everything and rebuild. Having things consistently work is much cheaper than down days caused by your ancient equipment. Apple’s SSDs will make it to 5 years no problem - and more likely, 10-15 years.
At my last N jobs, companies built high end server farms and carefully specced all the hardware. Then they looked at SSD specs and said “these are all fine”. Fast forward 2 years: The $50-$250K machines have a 100% drive failure rate, and some poor bastard has to fly from data center to data center to swap the $60 drive for a $120 one, then re-rack and re-image each machine. Anyway, soldering a decent SSD to the moth…
What does soldering it to the board have to do with reliability?
If they were soldered onto those systems you talk about, all those would have had to be replaced instead of just having the drive swapped out and re-imaged.
Re: Apple M3 Ultra
#464512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
It will cost 4X what it costs to get 512GB on an x86 server motherboard.
That’s not going to yield the same bandwidth or memory latency though, right?
Re: Apple M3 Ultra
#465Now with Ultra-class backdoors? https://news.ycombinator.com/item?id=43003230
Re: Apple M3 Ultra
#466Earlier quoted context omitted.
> The the question is if a llm will run with usable performance at that scale? This is the big question to have answered. Many people claim Apple can now reliably be used as a ML workstation, but from the numbers I've seen from benchmarks, the models may fit in memory, but the performance for tok/sec is so slow to not feel worth it, compared to running it on NVIDIA hardware. Although it be expensive as hell to get 51…
Do we know if is it slower because of hardware is not as well suited for the task or is it mostly a software issue -- the code hasn't been optimized to run on Apple Silicon?
AFAICT the neural engine has accelerators for CNNs and integer math, but not the exact tensor operations in popular LLM transformer architectures that are well-supported in GPUs.
Re: Apple M3 Ultra
#467Earlier quoted context omitted.
> Probably helps that models like deepseek are mixture of expert. Having all weights in VRAM means you don’t have to unlod/reload. Memory bandwidth usage should be limited to the 37B active parameters. "Memory bandwidth usage should be limited to the 37B active parameters." Can someone do a deep dive above quote. I understand having the entire model loaded into RAM helps with response times. However, I don't quite un…
With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.
What people who did not actually work with this stuff in practice don't realize is the above statement only holds for batch size 1, sequence size 1. For processing the prompt you will need to read all the weights (which isn't a problem, because prefill is compute-bound, which, in turn is a problem on a weak machine like this Mac or an "EPYC build" someone else mentioned). Even for inference, batch size greater than 1 (more than one inference at a time) or sequence size of greater than 1 (speculative decoding), could require you to read the entire model, repeatedly. MoE is beneficial, but there's a lot of nuance here, which people usually miss.
Re: Apple M3 Ultra
#468Earlier quoted context omitted.
Nah, if I ever wrote an article about the software crisis on the Linux desktop, there’d be flames here making Apple’s issues look small.
It'd be an interesting flame war in the comments, if nothing else, go for it! I'm happy to give plenty of concrete evidence why Linux is more suitable for professionals than macOS is in 2025 :)
Omg I despise the fact that there's n competing GUI standards on linux, zero visual consistency.
I love diversity in websites, and apps for that matter, but this isn't diversity, it is the uncanny valley between bespoke graphic design and homogeneity.
Say what you want about SwiftUI, but it makes consistent, good looking apps. Unless something has changed, GTK is a usability disaster.
And that's before I get into how much both X11 and wayland suck equally.
There's so much I miss about Linux, but there's so much I don't
Re: Apple M3 Ultra
#469512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
I think the other big thing is that the base model finally starts at a normal amount of memory for a production machine. You can't get less than 96GB. Although an extra $4000 for the 512GB model seems Tim Apple levels of ridiculous. There is absolutely no way that the different costs anywhere near that much at the fab.
And the storage solution still makes no sense of course, a machine like this should start at 4TB for $0 extra, 8TB for $500 more, and 16TB for $1000 more. Not start at a useless 1TB, with the 8TB version costing an extra $2400 and 16TB a truly idiotic $4600. If Sabrent can make and sell 8TB m.2 NVMe drives for $1000, SoC storage should set you back half that, not over double that.
Re: Apple M3 Ultra
#470Earlier quoted context omitted.
Apples to oranges. NVIDIA cards have an order of magnitude more horsepower for compute than this thing. A B100 has 8 TB/s of memory bandwidth, 10 times more than this. If NVIDIA made a card with 512GB of HBM I'd expect it to cost $150K. The compute and memory bandwidth of the M3 Ultra is more in-line with what you'd get from a Xeon or Epyc/Threadripper CPU on a server motherboard; it's just that the x86 "way" of doin…
Yep, it's apples to oranges. But sometimes you want apples, and sometimes you want oranges, so it's all good! There's a wide spectrum of potential requirements between memory capacity, memory bandwidth, compute speed, compute complexity, and compute parallelism. In the past, a few GB was adequate for tasks that we assigned to the GPU, you had enough storage bandwidth to load the relevant scene into memory and generat…
> we still work with powers of two. Please.
We do. Common people don't. It's easier to write "over half a terabyte" than explain (again) to millions of people what the power of two is.