Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

github.com

Post reply on HN
Print view View original

11–20 of 124 posts

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#11

Post by ryao »

superkuh wrote: post #8
Cheer2171 wrote: post #6

Earlier quoted context omitted.

You don't have to go that far down the page to see it is paging to system RAM: Requirements: 380GB CPU Memory 1-8 ARC A770 500GB Disk

Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorr…

What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#13

Post by superkuh »

ryao wrote: post #9
superkuh wrote: post #3

No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.

It is theoretically possible. Each token only needs 37B parameters and if the same experts are chosen often, it would behave closer to a 37B model than a 671B model, since reusing experts can skip loads from system RAM. You might still be right since I have not confirmed that the selected experts change infrequently doing prompt processing / token generation, and someone could have botched the headline. However, trea…

MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still spending a lot of money which makes this a lot less novel or interesting than "just on 1~2 Arc A770" implies. Especially for the marginal performance that even 8-12 channels of CPU memory bandwidth gets you.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#14

Post by zamadatix »

rgbrgb wrote: post #5
superkuh wrote: post #3

No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.

yep, title is inaccurate. it's a distill into Qwen 7B DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf

The document contains multiple sections. The initial section does reference DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf as the example model but if you continue reading further you'll see a section referencing running DeepSeek-R1-Q4_K_M.gguf plus claims several other variations have been tested.

It's a bit less exciting when you see they're just talking about offloading parts from the large amount of DRAM.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#15

Post by zamadatix »

Since the Xeon alone could run the model in this set up it'd be more interesting if they compared the performance uplift with using 0/1/2..8 Arc A770 GPUs.

Also, it's probably better to link straight to the relevant section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#17

Post by colorant »

zamadatix wrote: post #15

Since the Xeon alone could run the model in this set up it'd be more interesting if they compared the performance uplift with using 0/1/2..8 Arc A770 GPUs. Also, it's probably better to link straight to the relevant section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...

Yes, you are right. Unfortunately HN somehow truncated my original URL link.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#18

Post by zamadatix »

colorant wrote: post #17
zamadatix wrote: post #15

Since the Xeon alone could run the model in this set up it'd be more interesting if they compared the performance uplift with using 0/1/2..8 Arc A770 GPUs. Also, it's probably better to link straight to the relevant section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...

Yes, you are right. Unfortunately HN somehow truncated my original URL link.

Sounds like submission "helper" tools are working about as well as normal :).

Did you have the chance to try this out yourself or did you just run across it recently?

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#20

Post by colorant »

rgbrgb wrote: post #5
superkuh wrote: post #3

No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.

yep, title is inaccurate. it's a distill into Qwen 7B DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf

See this section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...
Post reply on HN