471–480 of 1001 posts
Earlier quoted context omitted.
Every single AI shop on the planet is trying to figure out if there is enough compute or not to make this a reasonable AI path. If the answer is yes, that 10k is a absolute bargain.
Not much to figure out. It's 2x M4 Max, so you need 100 of these to match the TOPS of even a single consumer card like the RTX 5090.
Earlier quoted context omitted.
nobody in industry is using a 4090, they are using H100s which have 3TB/s. Apple also doesn’t have any equivalent to nvlink. I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops
4.8TB/s on H200, 8TB/s on B200, pretty insane.
Earlier quoted context omitted.
No native docker support, no headless management options (enterprise strength), Limited QoS management, lack of robust python support (out of the box), interactive user focused security model.
> No native docker support Honest question: why do you want this in MacOS? Do you understand what docker does? (it's fundamentally a linux technology, unless you are asking for user namespaces and chroot w/o SIP on MacOS, but that doesn't make sense since the app sandbox exists). MacOS doesn't have the fundamental ecosystem problems that beget the need for docker. If the answer is "I want to run docker containers bec…
Anyone wanting to run and manage their own suite of Macs to build multiple massive iOS and Mac apps at scale, for dozens or hundreds or thousands of developers deploying their changes.
xcodebuild is by far the most obvious "needs native for max perf" but there are a few other tools that require macOS. But obviously if you have multiple repos and apps, you might require many different versions of the same tools to build everything.
Sounds like a perfect use case for native containers.
512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…
If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.
I know it's basically nitpicking competing luxury sports cars at this point, but I am very bothered that existing benchmarks for the M3 show single core perf that is approximately 70% of M4 single core perf. I feel like I should be able to spend all my money to both get the fastest single core performance AND all the cores and available memory, but Apple has decided that we need to downgrade to "go wide". Annoying.
I'm a major Apple skeptic myself, but hasn't there always been a tradeoff between "fastest single core" vs "lots of cores" (and thus best multicore)?
For instance, I remember when you could buy an iMac with an i9 or whatever, with a higher clock speed and faster single core, or you could buy an iMac Pro with a Xeon with more cores, but the iMac (non-Pro) would beat it in a single core benchmark. Note: Though I used Macs as the example due to the simple product lines, I thought this was pretty much universal among all modern computers.
512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…
(and the 512GB version is $4,000 more rather than $10,000 - that's still worth mocking, but it's nowhere near as much)
Earlier quoted context omitted.
A beefy GPU which can't hold models in VRAM is of very limited use. You'll see 16 GB of VRAM on gamer Nvidia cards, the RTX 5090 being an exception with 32 GB VRAM. The professional cards have around 96 GB of VRAM. The thing with these Apple chips is that they have unified memory, where CPU and GPU use the same memory chips, which means that you can load huge models into RAM (no longer VRAM, because that doesn't exis…
which professional card has 96GB of VRAM?
https://www.primeline-solutions.com/de/nvidia-h100-nvl-94gb-...
Earlier quoted context omitted.
> The the question is if a llm will run with usable performance at that scale? This is the big question to have answered. Many people claim Apple can now reliably be used as a ML workstation, but from the numbers I've seen from benchmarks, the models may fit in memory, but the performance for tok/sec is so slow to not feel worth it, compared to running it on NVIDIA hardware. Although it be expensive as hell to get 51…
It is much slower than nVidia, but for a lot of personal-use LLM scenarios, it's very workable. And it doesn't need to be anywhere near as fast considering it's really the only viable (affordable) option for private, local inference, besides building a server like this, which is no faster: https://news.ycombinator.com/item?id=42897205