Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

twitter.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#11

Post by mrcwinn »

talldayo wrote: post #5
epistasis wrote: post #3

Earlier quoted context omitted.

That's a hell of a lot cheaper than running the equivalent H100 at home... And cheaper than a lot hobbyists' bicycles!

And only twice as expensive as the competing hardware you use to run R1 671B at 3-bit quantization! Ordinarily Apple customers cough up 3 or 4 times list price to match the performance of an equivalent PC. This is record-setting generosity from Cupertino.

Serious question coming from ignorance — what is the most cost effective way to run this locally, Mac or PC? Please, no fanboyism from either side. My understanding is that Apple's unified memory architecture is a leg up for that platform given the memory needs of these models, versus stringing together lots of NVidia GPUs.

Maybe I'm mistaken! Grateful to be corrected.

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#12

Post by OisinMoran »

doctoboggan wrote: post #8
shihab wrote: post #4

Please note that it’s using pretty aggressive quantization (around 4 bits per weight)

Its not that aggressive of a quantization considering that the full model was trained at only 8 bits.

That's 16x fewer possible values though (and also just 16 possible values full stop). It would be like giving every person on Earth the same shoe size.

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#14

Post by rashidae »

This is amazing!! What kind of applications are you considering for this? A part from saving variable costs, fine tuning extensively and security… I’m curious to evaluate this in a financial perspective, as variable costs can be daunting, but not too much “yet”.

I’m hoping NVIDIA comes up with their new consumer computer soon!

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#16

Post by wincy »

mrcwinn wrote: post #11
talldayo wrote: post #5

Earlier quoted context omitted.

And only twice as expensive as the competing hardware you use to run R1 671B at 3-bit quantization! Ordinarily Apple customers cough up 3 or 4 times list price to match the performance of an equivalent PC. This is record-setting generosity from Cupertino.

Serious question coming from ignorance — what is the most cost effective way to run this locally, Mac or PC? Please, no fanboyism from either side. My understanding is that Apple's unified memory architecture is a leg up for that platform given the memory needs of these models, versus stringing together lots of NVidia GPUs. Maybe I'm mistaken! Grateful to be corrected.

I think for $6000 you can run an EPYC setup. But the token/sec is going to be objectively slower than the Macs. What you gain on them is speed. I read this [0] on X earlier today which seems like a good guide on how to get yourself up and running.

[0] https://x.com/i/bookmarks/1884342681590960270?post_id=188424...

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#17

Post by mythz »

Someone also got the full Q8 R1 running on a $6K PC without a GPU on 2x EPYC with 768GB DDR5 RAM running at 6-8 tok/s [1].

Will be interesting to see the value/performance compared to next gen M4 Ultra's (or Extreme?) vs NVIDIA's new DIGITS [2] when they're released.

[1] https://x.com/carrigmat/status/1884244369907278106

[2] https://www.nvidia.com/en-us/project-digits/

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#19

Post by tetrisgm »

OisinMoran wrote: post #12
doctoboggan wrote: post #8

Earlier quoted context omitted.

Its not that aggressive of a quantization considering that the full model was trained at only 8 bits.

That's 16x fewer possible values though (and also just 16 possible values full stop). It would be like giving every person on Earth the same shoe size.

The Henry Ford school of product

Re: DeepSeek R1 671B running on 2 M2 Ultras faster than reading speed

#20

Post by shihab »

doctoboggan wrote: post #8
shihab wrote: post #4

Please note that it’s using pretty aggressive quantization (around 4 bits per weight)

Its not that aggressive of a quantization considering that the full model was trained at only 8 bits.

That doesn't necessarily mean final weights are 8-bit though. Tensor core ops are usually mixed precision- matmul happens in low precision but accumulation (i.e. final result) is done in much higher precision to reduce error.

from deepseek v3:

"For this reason, after careful investigations, we maintain the original precision (e.g., BF16 or FP32) for the following components: the embedding module, the output head, MoE gating modules, normalization operators, and attention operators...To further guarantee numerical stability, we store the master weights, weight gradients, and optimizer states in higher precision. "

Post reply on HN