[dead]
Apple M3 Ultra
521–530 of 1001 posts
Re: Apple M3 Ultra
#522Earlier quoted context omitted.
Nvidia has had the Grace Hoppers for a while now. Is this not like that?
This is cheap compared to GB200, which has a street price of >$70k for just the chip alone if you can even get one. Also GB200 technically has only 192GB per GPU and access to more than that happens over NVLink/RDMA, whereas here it’s just one big flat pool of unified memory without any tiered access topology.
We finally encountered the situation where an Apple computer is cheaper than its competition ;-)
All joking aside, I don't think Apples are that expensive compared to similar high-end gear. I don't think there is any other compact desktop computer with half a terabyte of RAM accessible to the GPU.
Re: Apple M3 Ultra
#523Earlier quoted context omitted.
Every single AI shop on the planet is trying to figure out if there is enough compute or not to make this a reasonable AI path. If the answer is yes, that 10k is a absolute bargain.
> that 10k is a absolute bargain The higher end NVidia workstation boxes won’t run well on normal 20amp plugs. So you need to move them to a computer room (whoops, ripped those out already) or spend months getting dedicated circuits run to office spaces.
In the US, normal circuits aren't always 20A, especially in residential buildings, where they are more commonly 15A in bedrooms and offices.
Re: Apple M3 Ultra
#524Re: Apple M3 Ultra
#525Earlier quoted context omitted.
I think the other big thing is that the base model finally starts at a normal amount of memory for a production machine. You can't get less than 96GB. Although an extra $4000 for the 512GB model seems Tim Apple levels of ridiculous. There is absolutely no way that the different costs anywhere near that much at the fab. And the storage solution still makes no sense of course, a machine like this should start at 4TB fo…
> There is absolutely no way that the different costs anywhere near that much at the fab. price premium probably, but chip lithography errors (thus, yields) at the huge memory density might be partially driving up the cost for huge memory.
It's Apple, price premium is a given.
Re: Apple M3 Ultra
#526Earlier quoted context omitted.
I torrent things from two different hosts on my gigabit network. The macos stack literally cannot handle the full bandwidth I have. It fails and the machine needs to be rebooted to fix it. It’s not pretty on the way into this state, either. Other remote connections to the computer are unreliable. On Linux, running the same app in a docker container works perfectly. Transmission is the app.
I get nearly 10Gbps from my NAS to my Mac Studio. It absolutely can handle that bandwidth. It may not handle that specific client well for unrelated reasons.
Bandwidth, yes. Connections count, no.
Re: Apple M3 Ultra
#527Earlier quoted context omitted.
> This hardware is really being held back by the operating system at this point. Apple could either create a 2U rack hardware and support Linux (and I mean Apple supporting it, not hobbysts), or have a build of Darwin headless that could run on that hardware. But in the later case, we probably wouldn't have much software available (though I am sure people would eventually starting porting software to it, there is alr…
There has to be someone at Apple with a contact at IBM that could make Fedora Apple Remix happen. It may not be on-brand, but this is a prime opportunity to make the competition look worse. File it under Community projects at https://opensource.apple.com/projects
Re: Apple M3 Ultra
#528Earlier quoted context omitted.
With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.
What people who did not actually work with this stuff in practice don't realize is the above statement only holds for batch size 1, sequence size 1. For processing the prompt you will need to read all the weights (which isn't a problem, because prefill is compute-bound, which, in turn is a problem on a weak machine like this Mac or an "EPYC build" someone else mentioned). Even for inference, batch size greater than 1…
No one should be buying this for batch inference obviously.
I remember right after OpenAI announced GPT3 I had a conversation with someone where we tried to predict how long it would be before GPT3 could run on a home desktop. This mac studio that has enough VRAM to run the full 175B parameter GPT3 with 16bit precision, and I think that’s pretty cool.
Re: Apple M3 Ultra
#529Earlier quoted context omitted.
If you're going to overthrow your entire AI workflow to use a different API anyway, surely the AMD Instinct accelerator cards make more sense. They're expensive, but also a lot faster, and you don't need to deal with making your code work on macOS.
I don't think API has any value because writing software is free and hardware for ML is super expensive.
I'd like to hire you to write free software
Re: Apple M3 Ultra
#530Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…
If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.
Doesn't MacOS support these things? I'm sure Apple runs these on their datacenters somehow?