Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

github.com

Post reply on HN
Print view View original

51–60 of 124 posts

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#51

Post by genewitch »

hmottestad wrote: post #34
genewitch wrote: post #31

Earlier quoted context omitted.

So you thought there was some magical way to get >600B parameters in a couple of GPUs? Also, LM studio lets you run smaller models in front of larger ones, so I could see having a few GPU in front really speeding up using R1 for inference.

The MoE architecture allows you to keep the entire active model on a single GPU. If two consecutive tokens use the same export then the second token is going to be much faster.

I understand all that, I am talking about a separate feature that is possibly backported or from llama.cpp. Where you have a small model that runs first and that is checked by a large model. I've seen 30%+ speedups using like 1.5B in front of a 15B for example.

Two GPUs or more mean you can start to "keep" one or more of the experts hot on a GPU as well.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#52

Post by vlovich123 »

yongjik wrote: post #39

Did DeepSeek learn how to name their models from OpenAI.

The convention is weird but it's pretty standard in the industry across all models, particularly GGUF. 671B parameters, quantized to 4 bits. The K_M terminology I believe is more specific to GGUF and describes the specific quantization strategy.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#53

Post by aurareturn »

colorant wrote: post #19
colorant wrote: post #16

https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic... Requirements (>8 token/s): 380GB CPU Memory 1-8 ARC A770 500GB Disk

Also see the demo from Jason Dai's post: https://www.linkedin.com/posts/jasondai_with-the-latest-ipex...

CPU inference is both bandwidth and compute constrained.

If your prompt has 10 tokens, it’ll do ok, like in the LinkedIn demo. If you need to increase the context, compute bottleneck will kick in quickly.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#54

Post by aurareturn »

ynniv wrote: post #44
faizshah wrote: post #40

Earlier quoted context omitted.

Anyone got a rough estimate of the cost of this setup? I’m guessing it’s under 10k. I also didn’t see tokens per second numbers.

It better be! AMD @ $2k: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...

This article keeps getting posted but it runs a thinking model at 3-4 tokens/s. You might as well take a vacation if you ask it a question.

It’s a gimmick and not a real solution.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#55

Post by zamadatix »

utopcell wrote: post #50
hmottestad wrote: post #34

Earlier quoted context omitted.

The MoE architecture allows you to keep the entire active model on a single GPU. If two consecutive tokens use the same export then the second token is going to be much faster.

What is the probability of that happening?

DeepSeek V3/R1 uses 8 routed experts out of 256, so not all as often as one would like. That said, having even just a single GPU will greatly speed up prompt processing which is worth it even if the inference speed was the same.

Ktransformers has a document about using CPU + a single 4090D to reach decent tokens/s but I'm not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU side https://github.com/kvcache-ai/ktransformers/blob/main/doc/en... The final step of going to 6 experts instead of 8 feels like cheating (not a lossless optimization).

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#56

Post by hnuser123456 »

aurareturn wrote: post #54
ynniv wrote: post #44

Earlier quoted context omitted.

It better be! AMD @ $2k: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...

This article keeps getting posted but it runs a thinking model at 3-4 tokens/s. You might as well take a vacation if you ask it a question. It’s a gimmick and not a real solution.

If you value local compute and don't need massive speed, that's still twice as fast as most people can type.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#57

Post by genewitch »

zamadatix wrote: post #55
utopcell wrote: post #50

Earlier quoted context omitted.

What is the probability of that happening?

DeepSeek V3/R1 uses 8 routed experts out of 256, so not all as often as one would like. That said, having even just a single GPU will greatly speed up prompt processing which is worth it even if the inference speed was the same. Ktransformers has a document about using CPU + a single 4090D to reach decent tokens/s but I'm not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU sid…

where does 256 come from? it's repeated in here and elsewhere that a single expert is 37B sized, so you'd have to have way more than "several hundred billion parameters", to hold 256 of those? Maybe i don't understand the architecture, but if that's the case, then everyone repeating 37B doesn't, either.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#58

Post by walrus01 »

aurareturn wrote: post #54
ynniv wrote: post #44

Earlier quoted context omitted.

It better be! AMD @ $2k: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...

This article keeps getting posted but it runs a thinking model at 3-4 tokens/s. You might as well take a vacation if you ask it a question. It’s a gimmick and not a real solution.

It's meant to be a test/development setup for people to prepare the software environment and tooling for running the same on more expensive hardware. Not to be fast.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#59

Post by hnfong »

As other commenters have mentioned, the performance of this set up is probably not really great since there's not enough VRAM and lots of bits have to be moved between CPU and GPU RAM.

That said, there are sub-256GB quants of DeepSeek-R1 out there (not the distilled versions). See https://unsloth.ai/blog/deepseekr1-dynamic

I can't quantify the difference between these and the full FP8 versions of DSR1, but I've been playing with these ~Q2 quants and they're surprisingly capable in their own right.

Another model that deserves mention is DeepSeek v2.5 (which has "fewer" params than V3/R1) - but still needs aggressive quantization before it can run on "consumer" devices (with less than ~100GB VRAM), and this is recently done by a kind soul: https://www.reddit.com/r/LocalLLaMA/comments/1irwx6q/deepsee...

DeepSeek v2.5 is arguably better than Llama 3 70B, so it should be of interest to anyone looking to run local inference. I really think more people should know about this.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#60

Post by zamadatix »

genewitch wrote: post #57
zamadatix wrote: post #55

Earlier quoted context omitted.

DeepSeek V3/R1 uses 8 routed experts out of 256, so not all as often as one would like. That said, having even just a single GPU will greatly speed up prompt processing which is worth it even if the inference speed was the same. Ktransformers has a document about using CPU + a single 4090D to reach decent tokens/s but I'm not sure how much of the perf is due to the 4090D vs other optimizations/changes for the CPU sid…

where does 256 come from? it's repeated in here and elsewhere that a single expert is 37B sized, so you'd have to have way more than "several hundred billion parameters", to hold 256 of those? Maybe i don't understand the architecture, but if that's the case, then everyone repeating 37B doesn't, either.

I think this diagram from the DeepSeekMoE paper explains it the clearest: https://i.imgur.com/CRKttob.png The one on the right is how the feed forward layers of DeepSeek V3/R1 work, blue and green are experts, and everything in that right section is what counts as "active parameters".

K (K=8 for these models, but you can customize that if you want) experts of 256 per layer are activated at a time. The 256 comes from the model file, it's just how many they chose to build it with. In these models there is also 1 shared expert which is always active in the layer. The router picks which k routed experts to use each forward pass and then a gating mechanism combines the outputs. If you sum the 1 shared expert + K routed experts + router + output networks you end up with 37 B parameters active for each feed forward layer pass. The individual experts are therefore much smaller than the total (probably something like 4 B parameters each? I've never really checked that directly).

Or, for the short answer: "37 B is the active parameters of 9 experts + 'overhead', not the parameters of a single expert".

Post reply on HN