Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

25L Portable NV-linked Dual 3090 LLM Rig

reddit.com

Post reply on HN
Print view View original

71–80 of 126 posts

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#71

Post by nxobject »

jychang wrote: post #48
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

Yeah, this page seems to be not great for beginners and also useless for people with experience. A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pric... Also, this phrasing is concerning: > WARNING - these componen…

Are the Alibaba 4090s modded to reach 48GB VRAM? (I ask only to figure how why they're that cheap...)

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#72

Post by jgalt212 »

fkyoureadthedoc wrote: post #61
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

> The page is blocking cut & paste. Super annoying! I've been running Don't F* With Paste* for years for this https://chromewebstore.google.com/detail/dont-f-with-paste/n...

Interesting. I guess our content-based marketing pages need to move to canvas-based rendering. That's probably bum too. Straight to serving up jpgs.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#73

Post by dotnet00 »

Gracana wrote: post #68
jychang wrote: post #48

Earlier quoted context omitted.

Yeah, this page seems to be not great for beginners and also useless for people with experience. A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pric... Also, this phrasing is concerning: > WARNING - these componen…

$2.5k is about $1k more than you'd spend on a pair of 3090s, and people I know who've bought blower 4090s say they sound like hair driers.

Blowers are loud, but they're easier to pack together, particularly given how most motherboards don't seem to space their two slots sufficiently to accomodate the massive coolers on recent GPUs.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#74

Post by Aurornis »

jychang wrote: post #48
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

Yeah, this page seems to be not great for beginners and also useless for people with experience. A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pric... Also, this phrasing is concerning: > WARNING - these componen…

Don’t those modified cards require hacked drivers? I would not want my expensive video card to depend on hacked drivers that may or may not continue to be available with new updates.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#75

Post by dotnet00 »

suladead wrote: post #17

I built pretty much this exact rig myself, but now it's gathering dust, any other uses for this rather than localLLMS

The 3090 I have in my server (Ollama on it is only used occasionally nowadays since I have dual 5080s on my work desktop), also handles accelerating transcoding in Plex, and is in the process of being setup to handle monitoring my 3d printers for failures via camera.

Am also considering setting up Home Assistant with LLM support again.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#76

Post by aunty_helen »

ismailmaj wrote: post #56
vladgur wrote: post #14

Earlier quoted context omitted.

I am exploring options just for fun. a used 3090 is around $900 on ebay. a used rtx 6000 ADA is around $5k 4 3090s are slower at inference and worse at training than 1 rtx 6000. 4x3090 would consume 1400W at load. Rtx 6000 would consume 300W at load. If you god forbid live in California and your power averages 45 cents per kwh, 4x3090 would be $1500+ more per year to operate than a single RTX 6000[0] [0] Back of the…

To make matters worse, the RTX3090 was released during the crypto craze and so a decent amount of the second hand market could contain overused GPUs that won’t last long, even if 3xxx to 4xxx performance difference is not that high, I would avoid the 3xxx series totally for resell value.

I bought 2 ex mining 3090s ~3 years ago. They’re in an always on pc that I remote into. Haven’t had a problem. If there was mass failures of gpus due to mining I would expect to have heard more about it

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#77

Post by ericdotlee »

Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

Any reason you wouldn't opt for the 4090 or 5090?

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#78

Post by tacomagick »

jychang wrote: post #48
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

Yeah, this page seems to be not great for beginners and also useless for people with experience. A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pric... Also, this phrasing is concerning: > WARNING - these componen…

Simply replacing the 3090's with 4090's would provide a major performance uplift assuming your model fits. (I have rented both 3090 and 4090 systems online for research, this comment is based on my personal experience, it is well worth the price increase and the hourly rate for the inference speed you get)

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#79

Post by ericdotlee »

jacquesm wrote: post #12
fxtentacle wrote: post #10

Earlier quoted context omitted.

The 3090 are a sweet spot for training. It’s the first generation with seriously fast VRAM. And it’s the last generation before Nvidia blocked NVlink. If you need to copy parameters between GPUs during training, the 3090 can be up to 70% faster than 4090 or 5090. Because the latter two are limited by PCI express bandwidth.

To be fair though, the 4090 and 5090 are much easier capable of saturating PCI express than the 3090 is, even at 4 lanes per card the 3090 rarely manages to saturate the links, it still handsomely pays off to split down to 4 lanes and add more cards. I used: https://c-payne.com/ Very high quality and manageable prices.

I've purchase 16 of these - cpayne is great! Hope he finds a US distributor to help with tariffs a bit!

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#80

Post by ericdotlee »

elsombrero wrote: post #55
imiric wrote: post #50

Earlier quoted context omitted.

I have a similar setup as the author with 2x 3090s. The issue is not that it's slow. 20-30 tk/s is perfectly acceptable to me. The issue is that the quality of the models that I'm able to self-host pales in comparison to that of SOTA hosted models. They hallucinate more, don't follow prompts as well, and simply generate overall worse quality content. These are issues that plague all "AI" models, but they are particul…

On my 2x 3090s I am running glm4.5 air q1 and it runs at ~300pp and 20/30 tk/s works pretty well with roo code on vscode, rarely misses tool calls and produces decent quality code. I also tried to use it with claude code with claude code router and it's pretty fast. Roo code uses bigger contexts, so it's quite slower than claude code in general, but I like the workflow better. this is my snippet for llama-swap ``` mo…

What is llama-swap?

Been looking for more details about software configs on https://llamabuilds.ai

Post reply on HN