Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

25L Portable NV-linked Dual 3090 LLM Rig

reddit.com

Post reply on HN
Print view View original

41–50 of 126 posts

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#41

Post by bedstefar »

cl3misch wrote: post #38

> The workplace of the coworker I built this for is truly offline, with no potential for LAN or wifi, so to download new models and update the system periodically I need to go pick it up from him and take it home. I'm surprised that a "truly offline" workplace allows servers to be taken home and being connected to the internet.

I worked in the Arctic for the better part of a decade. There's Starlink now, but I've been TRULY OFFLINE for weeks (with plenty diesel generated power) as recently as 2018. Technically we could use Iridium at like $10 per MB, but my full Wikipedia mirror (+ Debian/Ubuntu packages, PyPI etc) did come in handy more than once.

I know some Antarctic research stations (like McMurdo for example) still have connectivity restrictions depending on time-of-day, and I wouldn't be surprised if they also had mirrors of these sort of things, and/or dual-3090 rigs for llama.cpp in the off hours.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#42

Post by lifeinthevoid »

I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared.

I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#43

Post by nenenejej »

lifeinthevoid wrote: post #42

I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared. I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.

Graphics cards are so expensive (list price) they are cheap (no depreciation liquid market)

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#44

Post by prmoustache »

rtaylorgarlock wrote: post #33
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

[flagged]

I have js enabled and I can copy text on this page.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#45

Post by DecentShoes »

rtaylorgarlock wrote: post #33
Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

[flagged]

Horrible comment and attitude. People are trying to quote you for legitimate comment and criticism. This alone was enough for me to close the tab with your blog and ignore anything else you're going to say.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#46

Post by Tepix »

prmoustache wrote: post #44
rtaylorgarlock wrote: post #33

Earlier quoted context omitted.

[flagged]

I have js enabled and I can copy text on this page.

In general I can too, but try copying items from the "key specifications". Or perhaps I just had the impression because you can't mark text because I can't tell which text is marked and which isn't when marking text under "Key Specifications". Mea culpa.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#47

Post by gambiting »

DecentShoes wrote: post #45
rtaylorgarlock wrote: post #33

Earlier quoted context omitted.

[flagged]

Horrible comment and attitude. People are trying to quote you for legitimate comment and criticism. This alone was enough for me to close the tab with your blog and ignore anything else you're going to say.

That's not the author(I don't think?) just a random troll

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#48

Post by jychang »

Tepix wrote: post #29

OK, here's my quick critique of the article (having built a similar AM4-based system in 2023 for 2300€): 1) [I thought] The page is blocking cut & paste. Super annoying! 2) The exact mainboard is not specified exactly. There are 4 different boards called "ASUS ROG Strix X670E Gaming" and some of them only have one PCIe x16 slot. None of them can do PCIe x8 when using two GPUs. 3) The shopping link for the mainboard l…

Yeah, this page seems to be not great for beginners and also useless for people with experience.

A 2x 3090 build is okay for inference, but even with nvlink you're a bit handicapped for training. You're much better off with getting a 4090 48GB from China for $2.5k and just using that. Example: https://www.alibaba.com/trade/search?keywords=4090+48gb&pric...

Also, this phrasing is concerning:

> WARNING - these components don't fit if you try to copy this build. The bottom GPU is resting on the Arctic p12 slim fans at the bottom of the case and pushing up on the GPU. Also the top arctic p14 Max fans don't have mounting points for half of their screw holes, and are in place by being very tightly wedged against the motherboard, case, and PSU. Also, there's probably way too much pressure on the pcie cables coming off the gpus when you close the glass.

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#49

Post by justinclift »

vladgur wrote: post #35
logicallee wrote: post #15

Earlier quoted context omitted.

>I am exploring options just for fun. Since you're exploring options just for fun, out of curiosity, would you rent it out whenever you're not using it yourself, so it's not just sitting idle? (Could be noisy and loud). You'd be able to use your computer for other work at the same time and stop whenever you wanted to use it yourself.

It depends. At my electricity cost, 1 hour of 3090 or 1 hour of Rtx 6000 would cost the same 0.45 Just checked vast.ai. I will be losing money with 3090 at my electricity cost and making a tiny bit with rtx 6000. Like with boats it’s probably better to rent GPUs then buy them

Would a solar panel setup be an option for fixing that? :)

Re: 25L Portable NV-linked Dual 3090 LLM Rig

#50

Post by imiric »

lifeinthevoid wrote: post #42

I built a similar system, meanwhile I've sold one of the RTX 3090's. Local inference is fun and feels liberating, but it's also slow, and once I was used to the immense power of the giant hosted models, the fun quickly disappeared. I've kept a single GPU to still be able to play a bit with light local models, but not anymore for serious use.

I have a similar setup as the author with 2x 3090s.

The issue is not that it's slow. 20-30 tk/s is perfectly acceptable to me.

The issue is that the quality of the models that I'm able to self-host pales in comparison to that of SOTA hosted models. They hallucinate more, don't follow prompts as well, and simply generate overall worse quality content. These are issues that plague all "AI" models, but they are particularly evident on open weights ones. Maybe this is less noticeable on behemoth 100B+ parameter models, but to run those I would need to invest much more into this hobby than I'm willing to do.

I still run inference locally for simple one-off tasks. But for anything more sophisticated, hosted models are unfortunately required.

Post reply on HN