Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

github.com

Post reply on HN
Print view View original

121–124 of 124 posts

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#121

Post by genewitch »

numpad0 wrote: post #119
genewitch wrote: post #118

Earlier quoted context omitted.

I had a couple of those boards. The full size slots aren't actually x8, because the CPU those boards support only have 24 PCIe lanes, and over half of them are just running USB, SATA, etc. The full size slots are so you can secure cards to the board, instead of running them zip tied to metal dish racks (like in the video i linked in my reply to your sibling comment: https://www.youtube.com/shorts/rTCInAXSzKA )

No, there are versions of those with recycled Xeon E5 and bunch of x8(or so advertised) slots unlike most LGA115x mining boards.

sorry, i completely missed "xeon" in your comment.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#122

Post by superkuh »

ryao wrote: post #11
superkuh wrote: post #8

Earlier quoted context omitted.

Yep. That's why the headline is incorrect. 380GB of the model on CPU system RAM and 32GB on some ARC GPUs. The ratio, 380/32, is obvious. Most of the processing is being done on the CPU. The GPU are little bit icing in this context. Fast, sure, but having to wait for the CPU layers (that's how layer splits work with llama.cpp). I think changing the end of headline to "Xeon w/380GB RAM" would stop it from being incorr…

What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.

That's not how llama.cpp works. It's a layer split. The GPUs handle a few layers and the CPU handles the rest. The GPU layers no matter how fast they complete still have to wait on the CPU layers.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#123

Post by colorant »

superkuh wrote: post #122
ryao wrote: post #11

Earlier quoted context omitted.

What if it does not need to read from system RAM for every token by reusing experts whenever they just happen to be in VRAM from being used for the previous token? If the selected experts do not change often, this is doable on paper.

That's not how llama.cpp works. It's a layer split. The GPUs handle a few layers and the CPU handles the rest. The GPU layers no matter how fast they complete still have to wait on the CPU layers.

The ipex-llm implementation extends llama.cpp and includes additonal CPU-GPU hybrid optimizations for sparse MoE

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#124

Post by colorant »

moffkalast wrote: post #98
colorant wrote: post #73

Earlier quoted context omitted.

Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)

Decoding speed won't matter one bit if you have to sit there for 5 minutes waiting for the model to ingest a prompt that's two sentences long.

With ~1000 input, the TTFT is ~10 seconds
Post reply on HN