Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

github.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#111

Post by zargon »

npodbielski wrote: post #95
pshirshov wrote: post #93

Earlier quoted context omitted.

Ollama works fine with multi-gpu setups. Since rocm 6.3 everything is stable and you can mix different GPU generations. The performance is good enough for the models to be useful. The only thing which doesn't work well is running on iGPUs. It might work but it's very unstable.

Good to know. Still, is it viable option? Buying i.e. AMD Threadripper for 2.5k$, motherboard and ram for 2k$ and i.e. 4 GPUs for 4k$ to have total of 96GB of VRAM? Total should be around 10k$ which is roughly price of Intel GPU specifically for AI if I am not mistaken? Which option would be better performance wise. I did never saw a comparison anywhere and this is too much money to make fun experiment over the weeke…

> 10k$ which is roughly price of Intel GPU specifically for AI

Huh? The largest vram card that Intel has is the A770 which is around $350. What exactly are you trying to compare against? Are you doing inference only or training?

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#112

Post by genewitch »

zargon wrote: post #111
npodbielski wrote: post #95

Earlier quoted context omitted.

Good to know. Still, is it viable option? Buying i.e. AMD Threadripper for 2.5k$, motherboard and ram for 2k$ and i.e. 4 GPUs for 4k$ to have total of 96GB of VRAM? Total should be around 10k$ which is roughly price of Intel GPU specifically for AI if I am not mistaken? Which option would be better performance wise. I did never saw a comparison anywhere and this is too much money to make fun experiment over the weeke…

> 10k$ which is roughly price of Intel GPU specifically for AI Huh? The largest vram card that Intel has is the A770 which is around $350. What exactly are you trying to compare against? Are you doing inference only or training?

It can be read as a typo for "Nvidia" as their cards are about that price (or more, i haven't looked)

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#113

Post by refulgentis »

xoranth wrote: post #76
refulgentis wrote: post #66

Earlier quoted context omitted.

> It's the same thing here. CPUs can run it but only as a gimmick. No, that's not true. I work on local inference code via llama.cpp, on both GPU and CPU on every platform, and the bottleneck is much more ram / bandwidth than compute. Crappy Pixel Fold 2022 mid-range Android CPU gets you roughly same speed as 2024 Apple iPhone GPU, with Metal acceleration that dozens of very smart people hack on. Additionally, and pe…

> Crappy Pixel Fold 2022 mid-range Android CPU Can you share what LLMs do you run on such small devices/what user case they address? (Not a rhetorical question, it's just that I see a lot of work on local inference for edge devices with small models, but I could never get a small model to work for me. So I'm curious about other people's user cases.)

Excellent and accurate q. You sound like the first person I've talked to who might appreciate full exposition here, apologies if this is too much info. TL;DR is you're def not missing anything, and we're just beginning to turn a corner and see some rays of light of hope, where it's a genuine substitute for remote models in consumer applications.

#1) I put a lot of effort into this and, quite frankly, it paid off absolutely 0 until recently.

#2) The "this" in "I put a lot of effort into this", means, I left Google 1.5 years ago and have been quietly building an app that is LLM-agnostic in service of coalescing a lot of nextgen thinking re: computing I saw that's A) now possible due to LLMs B) was shitcanned in 2020, because Android won politically, because all that next-gen thinking seemed impossible given it required a step change in AI capabilities.

This app is Telosnex (telosnex.com).

I have a couple stringent requirements I enforce on myself, it has to run on every platform, and it has to support local LLMs just as well as paid ones.

I see that as essential for avoiding continued algorithmic capture of the means of info distribution, and believe on a long enough timeline, all the rushed hacking people have done to llama.cpp to get model after model supported will give away to UX improvements.

You are completely, utterly, correct to note that the local models on device are, in my words, useless toys, at best. In practice, they kill your battery and barely work.

However, things did pay off recently. How?

#1) llama.cpp landed a significant opus of a PR by @ochafik that normalized tool handling across models, as well as implemented what the models need individually for formatting

#2) Phi-4 mini came out. Long story, but tl;dr: till now there's been various gaping flaws with each Phi release. This one looked absent of any issues. So I hack support for its tool vagaries on top of what @ochafik landed, and all of a sudden I'm seeing the first local model sub-Mixtral 8x7B that's reliably handling RAG flows (i.e. generate search query, then, accept 2K tokens of parsed web pages and answer a q following directions I give you) and tool calls (i.e. generate search query, or file operations like here: https://x.com/jpohhhh/status/1897717300330926109)

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#114

Post by genewitch »

walrus01 wrote: post #84
numpad0 wrote: post #75

Earlier quoted context omitted.

PowerEdge R series is significantly cheaper if you already have an ear protection

yes, an R730 or R740 for instance. There's lots of used R630 and R640 with 512GB of RAM as well, but a 1U server is not the best thing to try putting gaming GPU type pci-express video cards into.

i imagine you can't use the pcie 1x risers on inference workloads, like you can for crypto mining, as there's no data going between the card and the CPU in crypto and i guess inference is heavily bandwidth restricted. Unfortunate!

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#115

Post by walrus01 »

genewitch wrote: post #114
walrus01 wrote: post #84

Earlier quoted context omitted.

yes, an R730 or R740 for instance. There's lots of used R630 and R640 with 512GB of RAM as well, but a 1U server is not the best thing to try putting gaming GPU type pci-express video cards into.

i imagine you can't use the pcie 1x risers on inference workloads, like you can for crypto mining, as there's no data going between the card and the CPU in crypto and i guess inference is heavily bandwidth restricted. Unfortunate!

The PCI-E slots in something like an 1U R630 or R640 are x8 or x16, the problem is more with the cooling and size/shape of the cards and how they cool themselves. The slots are meant to be used with things like 10 or 100 Gbps network cards or SAS/SATA host adapters which are considerably lower wattage than even a video card that's much weaker than an intel 770.

Commonly you will also find configuration with two or three 'low profile' pci-express slots which have a different card height than the 'standard' height that most GPUs are built at.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#116

Post by genewitch »

walrus01 wrote: post #115
genewitch wrote: post #114

Earlier quoted context omitted.

i imagine you can't use the pcie 1x risers on inference workloads, like you can for crypto mining, as there's no data going between the card and the CPU in crypto and i guess inference is heavily bandwidth restricted. Unfortunate!

The PCI-E slots in something like an 1U R630 or R640 are x8 or x16, the problem is more with the cooling and size/shape of the cards and how they cool themselves. The slots are meant to be used with things like 10 or 100 Gbps network cards or SAS/SATA host adapters which are considerably lower wattage than even a video card that's much weaker than an intel 770. Commonly you will also find configuration with two or th…

no, i mean like this https://www.youtube.com/shorts/rTCInAXSzKA

that works for crypto because all the CPU sends to the card is the target ledger sha256sum (simplified) and the GPU generates nonces until the `sha256sum(sha256sum(nonce += ledger sum)` has however many zeros in the front. So until a card finds the correct nonce, or the server sends "new work" - a new ledger shasum, there's no traffic, really, between the GPU and the CPU. housekeeping, whatever, but not like 1GB/s!

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#117

Post by numpad0 »

genewitch wrote: post #114
walrus01 wrote: post #84

Earlier quoted context omitted.

yes, an R730 or R740 for instance. There's lots of used R630 and R640 with 512GB of RAM as well, but a 1U server is not the best thing to try putting gaming GPU type pci-express video cards into.

i imagine you can't use the pcie 1x risers on inference workloads, like you can for crypto mining, as there's no data going between the card and the CPU in crypto and i guess inference is heavily bandwidth restricted. Unfortunate!

There are Xeon-based crypto motherboards on AliExpress with ~five x8 slots, I do sometimes wonder if those would work. NVIDIA Tesla K80/P40 24GB are on eBay for minimum $15 apiece. 120GB VRAM under $500 or so. Or maybe you could theoretically do a 96GB per node cluster with one x8 link for bottleneck free interconnect.

But it's likely never going to work, too many driver, compatibility, requisite Kernel development, and power issues, to name a few. Probably cheaper in the end to just go buy 5090 and rant about CUDA.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#118

Post by genewitch »

numpad0 wrote: post #117
genewitch wrote: post #114

Earlier quoted context omitted.

i imagine you can't use the pcie 1x risers on inference workloads, like you can for crypto mining, as there's no data going between the card and the CPU in crypto and i guess inference is heavily bandwidth restricted. Unfortunate!

There are Xeon-based crypto motherboards on AliExpress with ~five x8 slots, I do sometimes wonder if those would work. NVIDIA Tesla K80/P40 24GB are on eBay for minimum $15 apiece. 120GB VRAM under $500 or so. Or maybe you could theoretically do a 96GB per node cluster with one x8 link for bottleneck free interconnect. But it's likely never going to work, too many driver, compatibility, requisite Kernel development ,…

I had a couple of those boards. The full size slots aren't actually x8, because the CPU those boards support only have 24 PCIe lanes, and over half of them are just running USB, SATA, etc. The full size slots are so you can secure cards to the board, instead of running them zip tied to metal dish racks (like in the video i linked in my reply to your sibling comment: https://www.youtube.com/shorts/rTCInAXSzKA)

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#119

Post by numpad0 »

genewitch wrote: post #118
numpad0 wrote: post #117

Earlier quoted context omitted.

There are Xeon-based crypto motherboards on AliExpress with ~five x8 slots, I do sometimes wonder if those would work. NVIDIA Tesla K80/P40 24GB are on eBay for minimum $15 apiece. 120GB VRAM under $500 or so. Or maybe you could theoretically do a 96GB per node cluster with one x8 link for bottleneck free interconnect. But it's likely never going to work, too many driver, compatibility, requisite Kernel development ,…

I had a couple of those boards. The full size slots aren't actually x8, because the CPU those boards support only have 24 PCIe lanes, and over half of them are just running USB, SATA, etc. The full size slots are so you can secure cards to the board, instead of running them zip tied to metal dish racks (like in the video i linked in my reply to your sibling comment: https://www.youtube.com/shorts/rTCInAXSzKA )

No, there are versions of those with recycled Xeon E5 and bunch of x8(or so advertised) slots unlike most LGA115x mining boards.

Re: DeepSeek-R1-671B-Q4_K_M with 1 or 2 Arc A770 on Xeon

#120

Post by ltbarcly3 »

manmal wrote: post #78
ltbarcly3 wrote: post #63

Earlier quoted context omitted.

The Q8 model is totally different?

My experience with quantizations is that anything below 6 is noticeably worse. Coherence suffers. I’ve rarely gotten anything really useful out of a Q4 model, code wise. For transformations they are great though, eg convert JSON to Markdown and vice versa.

No I mean the quantized versions of this model in particular have less parameters as well. They are almost different models.
Post reply on HN