What exactly does the Xeon do in this situation, is there a reason you couldn't use any other x86 processor?
DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
21–30 of 124 posts
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#22What exactly does the Xeon do in this situation, is there a reason you couldn't use any other x86 processor?
I think it’s that most non Xeon motherboards don’t have the memory channels to have this much memory with any sort of commercially viable dimms.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#23[deleted]
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#24I’ve been following the progress Intel Arc support in Pytorch is making, at least in Linux, and it seems like if things stay on track, we may see the first version of pytorch with full Xe/Arc support by around June. I think I’m just going to wait until then instead of dealing with anything ipex or openvino.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#25[deleted]
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#26What exactly does the Xeon do in this situation, is there a reason you couldn't use any other x86 processor?
There's not much else (other than Epyc) in the way of affordably priced motherboards that have enough cumulative RAM. You can buy a used Dell dual socket older xeon CPU server with 512GB of RAM for test/development purposes for not very much money.
Under $1500 (before adding video cards or your own SSD), easily, with what I just found in a few minutes of searching. I'm also seeing things with 1024GB of RAM for under $2000.
You also want to have the capability for more than one full speed at minimum PCI-Express x16 3.0 card, which means you need enough PCI-E lanes, which you aren't going to find on a single socket Intel workstation motherboard.
Here's a couple of somewhat randomly chosen examples with 512GB of RAM and affordably priced. they'll be power hungry, and noisy. Same general idea from other x86-64 hardware such as from hp, supermicro, etc. These are fairly common in quantity so I'm using them as a baseline for specification vs price. Configurations will be something with 16 x 32GB DDR4 DIMMs.
https://www.ebay.com/itm/186991103256?_skw=dell+poweredge+t6...
https://www.ebay.com/itm/235978320621?_skw=dell+poweredge+r7...
https://www.ebay.com/itm/115819389940?_skw=dell+poweredge+r7...
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#27Since the Xeon alone could run the model in this set up it'd be more interesting if they compared the performance uplift with using 0/1/2..8 Arc A770 GPUs. Also, it's probably better to link straight to the relevant section https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic...
If you’re running just one GPU your context is limited to 1024 tokens, as far as I could tell. I couldn’t see what the context size is for more cards though.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#28Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#29Earlier quoted context omitted.
Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorr…
What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.
That’s probably the main performance benefit of using the GPU. If you’re changing the active expert for every single token then it wouldn’t be any faster than just running it on the CPU. Once you can reuse the active expert for two tokens you’re already going to be a lot faster than just the CPU.
More GPUs let you keep more experts active at a time.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#30Earlier quoted context omitted.
Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorr…
What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.
Expert distribution should be approximately random token-by-token, so not likely.