Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

25L Portable NV-linked Dual 3090 LLM Rig

reddit.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#111

Post by Tepix »

yjftsjthsd-h wrote: post #92
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

> 3) The shopping link for the mainboard leads to the "ASUS ROG Strix X670E-E Gaming" model. This model can use the 2nd PCIe 5.0 port at only x4 speeds. The RTX 3090 can only do PCIe 4.0 of course so it will run at PCIe 4.0 x4. If you choose a desktop mainboard for having two GPUs, make sure it can run at PCIe x8 speeds when using both GPU slots! Having NVLink between the GPUs is not a replacement for having a fast c…

In general, if all you do is inference with a model that’s in VRAM, you’re right. OTOH it’s simply a matter of picking the right mainboard. If you have one of those sweet new MoE models that won‘t completely fit in your VRAM, offloading means you want PCIe bandwidth, because it will be a bottleneck. Also swapping between LLMs will be faster.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#112

Post by Our_Benefactors »

nenenejej wrote: post #43
lifeinthevoid wrote: post #42

I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared. I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.

Graphics cards are so expensive (list price) they are cheap (no depreciation liquid market)

Did you really claim GPUs have zero depreciation? That’s obviously false.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#113

Post by Our_Benefactors »

tayo42 wrote: post #16
jszymborski wrote: post #6

I just don't get why the RTX 4090 is still so expensive on the used market. New Rtx 5090s are almost as expensive!

Are these just for ai now? Or are games pushing video cards that much?

4090 is a great gaming card, the spiritual successor to the 1080. It will be viable for years and years.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#114

Post by imiric »

ThatPlayer wrote: post #60
imiric wrote: post #50

Earlier quoted context omitted.

I have a similar setup as the author with 2x 3090s. The issue is not that it's slow. 20-30 tk/s is perfectly acceptable to me. The issue is that the quality of the models that I'm able to self-host pales in comparison to that of SOTA hosted models. They hallucinate more, don't follow prompts as well, and simply generate overall worse quality content. These are issues that plague all "AI" models, but they are particul…

> behemoth 100B+ parameter models, but to run those I would need to invest much more into this hobby than I'm willing to do. Have you tried newer MoE models with llama.cpp's recent '--n-cpu-moe' option to offload MoE layers to the CPU? I can run gpt-oss-120b (5.1B active) on my 4080 and get a usable ~20 tk/s. Had to upgrade my system RAM, but that's easier. https://github.com/ggml-org/llama.cpp/discussions/15396 has…

I use Ollama which offloads to the CPU automatically IIRC. IME the performance drops dramatically when that happens, and it hogs the CPU making the system unresponsive for other tasks, so I try to avoid it.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#116

Post by chpatrick »

mycall wrote: post #51
imiric wrote: post #50

Earlier quoted context omitted.

I have a similar setup as the author with 2x 3090s. The issue is not that it's slow. 20-30 tk/s is perfectly acceptable to me. The issue is that the quality of the models that I'm able to self-host pales in comparison to that of SOTA hosted models. They hallucinate more, don't follow prompts as well, and simply generate overall worse quality content. These are issues that plague all "AI" models, but they are particul…

> 20-30 tk/s or ~2.2M tk/day. This is how we should be thinking about it imho.

Is it? If you're the only user then you care about latency more than throughput.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#117

Post by ThatPlayer »

imiric wrote: post #114
ThatPlayer wrote: post #60

Earlier quoted context omitted.

> behemoth 100B+ parameter models, but to run those I would need to invest much more into this hobby than I'm willing to do. Have you tried newer MoE models with llama.cpp's recent '--n-cpu-moe' option to offload MoE layers to the CPU? I can run gpt-oss-120b (5.1B active) on my 4080 and get a usable ~20 tk/s. Had to upgrade my system RAM, but that's easier. https://github.com/ggml-org/llama.cpp/discussions/15396 has…

I use Ollama which offloads to the CPU automatically IIRC. IME the performance drops dramatically when that happens, and it hogs the CPU making the system unresponsive for other tasks, so I try to avoid it.

I don't believe that's the same thing. That should be the generic offloading that ollama will do to any too big model, while this feature requires MoE models. https://github.com/ollama/ollama/issues/11772 is the feature request for similar on ollama.

One comment in that thread mentions getting almost 30tk/s from gpt-oss-120b on a 3090 with llama.cpp compared to 8tk/s with ollama.

This feature is limited to MoE models, but those seem to be gaining traction with gpt-oss, glm-4.5, and qwen3

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#118

Post by jgalt212 »

93po wrote: post #93
jgalt212 wrote: post #72

Earlier quoted context omitted.

Interesting. I guess our content-based marketing pages need to move to canvas-based rendering. That's probably bum too. Straight to serving up jpgs.

thankfully most web browsing will be done by LLMs soon and that won't stop them, good riddance to the mess of a web that google has created

dead Internet for realz

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#119

Post by mycall »

chpatrick wrote: post #116
mycall wrote: post #51

Earlier quoted context omitted.

> 20-30 tk/s or ~2.2M tk/day. This is how we should be thinking about it imho.

Is it? If you're the only user then you care about latency more than throughput.

Not if you have a queue of work that isn't a high priority, like edge compute to review changes in security cam footage or prepare my next day's tasks (calendar, commitments, needs, etc)

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#120

Post by imiric »

ThatPlayer wrote: post #117
imiric wrote: post #114

Earlier quoted context omitted.

I use Ollama which offloads to the CPU automatically IIRC. IME the performance drops dramatically when that happens, and it hogs the CPU making the system unresponsive for other tasks, so I try to avoid it.

I don't believe that's the same thing. That should be the generic offloading that ollama will do to any too big model, while this feature requires MoE models. https://github.com/ollama/ollama/issues/11772 is the feature request for similar on ollama. One comment in that thread mentions getting almost 30tk/s from gpt-oss-120b on a 3090 with llama.cpp compared to 8tk/s with ollama. This feature is limited to MoE models…

Ah, I was not aware of that, thanks. I'll give it a try.
Post reply on HN