Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

twitter.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#21

Post by [deleted] »

mrcwinn wrote: post #11
talldayo wrote: post #5

Earlier quoted context omitted.

And only twice as expensive as the competing hardware you use to run R1 671B at 3-bit quantization! Ordinarily Apple customers cough up 3 or 4 times list price to match the performance of an equivalent PC. This is record-setting generosity from Cupertino.

Serious question coming from ignorance — what is the most cost effective way to run this locally, Mac or PC? Please, no fanboyism from either side. My understanding is that Apple's unified memory architecture is a leg up for that platform given the memory needs of these models, versus stringing together lots of NVidia GPUs. Maybe I'm mistaken! Grateful to be corrected.

[deleted]

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#22

Post by rahimnathwani »

mythz wrote: post #17

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] https://x.com/carrigmat/status/1884244369907278106 [2] https://www.nvidia.com/en-us/project-digits/

Wow!

6 to 8 tokens per second.

And less than a tenth of the cost of a GPU setup.

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#23

Post by coder543 »

mrcwinn wrote: post #11
talldayo wrote: post #5

Earlier quoted context omitted.

And only twice as expensive as the competing hardware you use to run R1 671B at 3-bit quantization! Ordinarily Apple customers cough up 3 or 4 times list price to match the performance of an equivalent PC. This is record-setting generosity from Cupertino.

Serious question coming from ignorance — what is the most cost effective way to run this locally, Mac or PC? Please, no fanboyism from either side. My understanding is that Apple's unified memory architecture is a leg up for that platform given the memory needs of these models, versus stringing together lots of NVidia GPUs. Maybe I'm mistaken! Grateful to be corrected.

The most cost-effective way is arguably to run it off of any 1TB SSD (~$55) attached to whatever computer you already have.

I was able to get 1 token every 6 or 7 seconds (approximately 10 words per minute) on a 400GB quant of the model, while using an SSD that benchmarks at a measly 3GB/s or so. The bottleneck is entirely the speed of the SSD at that level, so an SSD that is twice as fast should make the model run about twice as fast.

Of course, each message you send would have approximately a 1 business day turnaround time… so it might not be the most practical.

With a RAID0 array of two PCIe 5.0 SSDs (~14GB/s each, 28GB/s total), you could potentially get things up to an almost tolerable speed. Maybe 1 to 2 tokens per second.

It’s just such an enormous model that your next best option is like $6000 of hardware, as another comment mentioned, and that is probably going to be significantly slower than the two M2 Ultra Mac Studios featured in the current post. It’s a sliding scale of cost versus performance.

This model has about half as many active parameters as Llama3-70B, since it has 37B active parameters, so it’s actually pretty easy to run computationally… but the catch is that you have to be able to access any 37B of those 671B parameters at any time, so you have to find somewhere fast to store the entire model.

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#24

Post by CamperBob2 »

mythz wrote: post #17

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] https://x.com/carrigmat/status/1884244369907278106 [2] https://www.nvidia.com/en-us/project-digits/

Alternative link: https://xcancel.com/carrigmat/status/1884244369907278106

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#25

Post by mrcwinn »

mythz wrote: post #17

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] https://x.com/carrigmat/status/1884244369907278106 [2] https://www.nvidia.com/en-us/project-digits/

Digits will be $3k and have 128GB of unified memory, so don't we already know that it wouldn't compare well this this rig? 128 won't be enough to fit the model in memory.

As for Apple, we'll see.

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#26

Post by phonon »

mythz wrote: post #17

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] https://x.com/carrigmat/status/1884244369907278106 [2] https://www.nvidia.com/en-us/project-digits/

Nice! Xeon 6 using AMX-BF16/INT8 Instructions should be something like 5 times faster than that....

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#27

Post by mythz »

mrcwinn wrote: post #25
mythz wrote: post #17

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] https://x.com/carrigmat/status/1884244369907278106 [2] https://www.nvidia.com/en-us/project-digits/

Digits will be $3k and have 128GB of unified memory, so don't we already know that it wouldn't compare well this this rig? 128 won't be enough to fit the model in memory. As for Apple, we'll see.

They can be linked, e.g. 2x DIGITS can run 405B models [1]. Won't know what value/performance we can get until they start shipping them in May.

https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe...

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#28

Post by danans »

Check out the power draw metrics. Following the CPU+GPU power consumption, it seems like it averaged 22W for about a minute. Unless I'm missing something, the inference for this example consumed at most .0004 kWh.

That's almost nothing. If these models are capable/functional enough for most day-to-day uses, then useful LLM-based GenAI is already at the "too cheap to meter" stage.

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#29

Post by fulafel »

mrcwinn wrote: post #25
mythz wrote: post #17

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1]. Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released. [1] https://x.com/carrigmat/status/1884244369907278106 [2] https://www.nvidia.com/en-us/project-digits/

Digits will be $3k and have 128GB of unified memory, so don't we already know that it wouldn't compare well this this rig? 128 won't be enough to fit the model in memory. As for Apple, we'll see.

What's the memory bandwidth like compared to the above EPYC setup that the tweeter claims has "24 channels of DDR5 RAM" ?
Post reply on HN