Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

731–740 of 1001 posts

Re: Apple M3 Ultra

#731

Post by jltsiren »

varjag wrote: post #39
klausa wrote: post #15

Earlier quoted context omitted.

It's over half a _tera_byte; exactly half of _tebi_byte if you wanna be a fundamentalist.

It is exactly the opposite. Every computer architecture in production addresses memory in the powers of two. SI has no business in memory size nomenclature as it is not derived from fundamental physical units. The whole klownbyte change was pushed through by hard drive marketers in 1990s.

It's more complicated than that. Data storage sizes are not connected to fundamental physical units, but data transfer rates are. Things get annoying when a 1 MB/s connection cannot transfer a megabyte in a second.

Re: Apple M3 Ultra

#732

Post by treesciencebot »

zitterbewegung wrote: post #420
valine wrote: post #382

Earlier quoted context omitted.

What would it cost to get 512GB of VRAM on an Nvidia card? That’s the real comparison.

Since the GH200 has over a terabyte of VRAM at $343,000 and the H100 has 80GB that makes that $195,993 with a bit over 512GB of VRAM . You could beat the price of the Apple M3 Ultra with an AMD EPYC build.

GH200 is nowhere near $343,000 number. You can get a single server order around 45k (with inception discount). If you are buying bulk, it goes down to sub-30k ish. This comes with a H100's performance and insane amount of high bandwith memory.

Re: Apple M3 Ultra

#733

Post by wmf »

ddtaylor wrote: post #636
samstave wrote: post #622

Earlier quoted context omitted.

"unified memory" funny that people think this is so new, when CRAY had Global Heap eons ago...

Why did it take so long for us to get here?

Laptops have had unified memory for ten years or more. For desktops very few apps benefit from unified memory.

Re: Apple M3 Ultra

#734

Post by wmf »

treesciencebot wrote: post #732
zitterbewegung wrote: post #420

Earlier quoted context omitted.

Since the GH200 has over a terabyte of VRAM at $343,000 and the H100 has 80GB that makes that $195,993 with a bit over 512GB of VRAM . You could beat the price of the Apple M3 Ultra with an AMD EPYC build.

GH200 is nowhere near $343,000 number. You can get a single server order around 45k (with inception discount). If you are buying bulk, it goes down to sub-30k ish. This comes with a H100's performance and insane amount of high bandwith memory.

They probably meant 8xH200 for $343,000 which is in the ballpark.

Re: Apple M3 Ultra

#735

Post by wmf »

muricula wrote: post #648
bmelton wrote: post #431

Earlier quoted context omitted.

https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...

Is that a CPU based inference build? Shouldn't you be able to get more performance out of the M3's GPU?

Inference is about memory bandwidth and some CPUs have just as much bandwidth as a GPU.

Re: Apple M3 Ultra

#736

Post by behnamoh »

DrBenCarson wrote: post #656
behnamoh wrote: post #628

Earlier quoted context omitted.

we’ve heard that claim for the past three years, but every effort by them points to the opposite. don’t get me wrong, I would love for Apple Intelligence to be smart enough on my iPhone and on my Mac, but honestly, the current version is a complete disappointment.

Apple are working on the hard problems of making AI useful (call them “agents”), not AGI 1. Small models running locally with well-established tool interfaces (“app intents”) 2. Large models running in a bespoke cloud that can securely and quickly load all relevant tokens from a device before running inference No AI lab is even close to what Apple is trying to deliver in the next ~12 months

if that were the case, then it would definitely help Apple Intelligence if the iPhone and Mac had higher amounts of RAM, but the base MacBook Pro announced by Apple a while ago had 8 GB of RAM and even the pro versions of the iPhone have 8GB whereas 12, 16, or even higher RAM is very common in android devices which helps users run relatively large language models on their devices

Re: Apple M3 Ultra

#737

Post by DidYaWipe »

How do people feel about the value of the M3 Ultra vs. the M4 Max for general computing, assuming that you max out the RAM on the M4 version of the Studio?

Re: Apple M3 Ultra

#738

Post by forrestthewoods »

lynguist wrote: post #718
lhl wrote: post #560

Earlier quoted context omitted.

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

I don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.

Re: Apple M3 Ultra

#739

Post by a1o »

lynguist wrote: post #718
lhl wrote: post #560

Earlier quoted context omitted.

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation. From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?),…

I actually think it’s not a coincidence and they specifically built this M3 Ultra for DeepSeek R1 4-bit. They also highlight in their press release that they tested it with 600B class LLMs (DeepSeek R1 without referring to it by name). And they specifically did not stop at 256 GB RAM to make this happen. Maybe I’m reading too much into it.

Any ideas on power consumption? I wonder how much power would that use. It looks like it would be more efficient than everything else that currently exists.

Re: Apple M3 Ultra

#740

Post by jonfromsf »

doctorpangloss wrote: post #492
ein0p wrote: post #467

Earlier quoted context omitted.

What people who did not actually work with this stuff in practice don't realize is the above statement only holds for batch size 1, sequence size 1. For processing the prompt you will need to read all the weights (which isn't a problem, because prefill is compute-bound, which, in turn is a problem on a weak machine like this Mac or an "EPYC build" someone else mentioned). Even for inference, batch size greater than 1…

Sure, nuance. This is why Apple makes so much fucking money: people will craft the wildest narratives about how they’re going to use this thing. It’s part of the aesthetics of spending $10,000. For every person who wants a solution to the problem of running a 400b+ parameter neural network, there are 19 who actually want an exciting experience of buying something, which is what Apple really makes. It has more in comm…

Birkin bags appreciate in value. This is more like a Lexus. It's a well-crafted luxury good that will depreciate relatively slowly.
Post reply on HN