Now we just need a model that can actually code
DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
41–50 of 124 posts
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#42Earlier quoted context omitted.
I think it’s that most non Xeon motherboards don’t have the memory channels to have this much memory with any sort of commercially viable dimms.
Pcie lanes
I was about to correct you because this doesn't use PCIe for anything, and then I realized Arc was a GPU (and they support up to 8 per machine).
Any idea how many Arc's it takes to match an H100?
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#43Article could stand to include a bit more information. Why are all the TPS figures x'ed out? What kind of performance can be expected from this setup (and how does it compare to the dual Epyc workstation recipe that was popularized recently?)
> the dual Epyc workstation recipe that was popularized recently Anyone have a link to this one?
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#44https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic... Requirements (>8 token/s): 380GB CPU Memory 1-8 ARC A770 500GB Disk
Anyone got a rough estimate of the cost of this setup? I’m guessing it’s under 10k. I also didn’t see tokens per second numbers.
It better be! AMD @ $2k: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#45Earlier quoted context omitted.
It is theoretically possible. Each token only needs 37B parameters and if the same experts are chosen often, it would behave closer to a 37B model than a 671B model, since reusing experts can skip loads from system RAM. You might still be right since I have not confirmed that the selected experts change infrequently doing prompt processing / token generation, and someone could have botched the headline. However, trea…
MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still spending a lot of money which makes this a lot less novel or interesting than "just on 1~2 Arc A770" implies. Especially for the marginal performance that even 8-12 channels of CPU memory bandwidth gets you.
Is this amount of RAM really that expensive? 6x 64GiB DDR4 DIMMs are < $1,000.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#46https://github.com/intel/ipex-llm/blob/main/docs/mddocs/Quic... Requirements (>8 token/s): 380GB CPU Memory 1-8 ARC A770 500GB Disk
[deleted]
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#47Earlier quoted context omitted.
It is theoretically possible. Each token only needs 37B parameters and if the same experts are chosen often, it would behave closer to a 37B model than a 671B model, since reusing experts can skip loads from system RAM. You might still be right since I have not confirmed that the selected experts change infrequently doing prompt processing / token generation, and someone could have botched the headline. However, trea…
MoE is pretty enabling after you've spent all the extra $$$$ to stuff your server CPU memory channels with ram so it's possible to run at all. But it's still spending a lot of money which makes this a lot less novel or interesting than "just on 1~2 Arc A770" implies. Especially for the marginal performance that even 8-12 channels of CPU memory bandwidth gets you.
Actually, 384GiB is already [1] https://www.amazon.com/NEMIX-RAM-DDR4-2666MHz-PC4-21300-Redu...
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#48Better to run the Q8 model on an epyc pair with 768GB, you'll get the same performance
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#49Earlier quoted context omitted.
Anyone got a rough estimate of the cost of this setup? I’m guessing it’s under 10k. I also didn’t see tokens per second numbers.
It better be! AMD @ $2k: https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...
What a teaser article! All this info for setting up the system, but no performance numbers.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#50Earlier quoted context omitted.
So you thought there was some magical way to get >600B parameters in a couple of GPUs? Also, LM studio lets you run smaller models in front of larger ones, so I could see having a few GPU in front really speeding up using R1 for inference.
The MoE architecture allows you to keep the entire active model on a single GPU. If two consecutive tokens use the same export then the second token is going to be much faster.
What is the probability of that happening?