DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
1–10 of 124 posts
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#2[deleted]
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#3No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model.
But a portable (no install) way to run llama.cpp on intel GPUs is really cool.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#4[deleted]
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#5No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.
yep, title is inaccurate. it's a distill into Qwen 7B DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#6No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.
You don't have to go that far down the page to see it is paging to system RAM:
Requirements:
380GB CPU Memory
1-8 ARC A770
500GB DiskRe: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#7No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.
[deleted]
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#8No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.
You don't have to go that far down the page to see it is paging to system RAM: Requirements: 380GB CPU Memory 1-8 ARC A770 500GB Disk
Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp).
I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorrect and misleading.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#9No... this headline is incorrect. You can't do that. I think they've confused the performance of running one of the small distills to existing smaller models. Two Arc cards cannot fit a 4 bit k-quant of a 671b model. But a portable (no install) way to run llama.cpp on intel GPUs is really cool.
It is theoretically possible. Each token only needs 37B parameters and if the same experts are chosen often, it would behave closer to a 37B model than a 671B model, since reusing experts can skip loads from system RAM.
You might still be right since I have not confirmed that the selected experts change infrequently doing prompt processing / token generation, and someone could have botched the headline. However, treating Deepseek like llama 3 when reasoning about VRAM requirements is not necessarily correct.
Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon
#10Earlier quoted context omitted.
You don't have to go that far down the page to see it is paging to system RAM: Requirements: 380GB CPU Memory 1-8 ARC A770 500GB Disk
Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorr…
"with" does not mean "entirely on"
Edit: but what you added in your edit is right, it would be more accurate to append the system ram requirement