Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

imil.net

Post reply on HN
Print view View original

111–116 of 116 posts

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#111

Post by arjie »

ydj wrote: post #110
arjie wrote: post #100

Earlier quoted context omitted.

Were you able to connect the two P150 using the qsfp-dd cable? They only sell 4x and 8x topologies so I’m curious if that worked for you. Are you able to run them tensor parallel?

Yeah, I’m doing TP with two cards. The topology is configured based on yaml files, and if you are not using a predefined config you can just create a new config with your topology. I’m not even using a 800G cable since they are expensive and I don’t think I need the bandwidth, opting for 400G instead. This just needs a config change for the number of Ethernet links it uses internally. (Apparently these cables are jus…

Brilliant, thank you. Maybe I'll get a couple in a bit.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#113

Post by oneshtein »

redox99 wrote: post #99
otabdeveloper4 wrote: post #97

Earlier quoted context omitted.

Number of parameters doesn't make the model smarter, it just makes it know more stuff out of the box. At some point there's diminishing returns and your coding LLM performs worse because you encoded useless stuff like Pokemon combinations or languages you don't speak into its parameter space. The "smartness" of the model comes from RLHF post-training, which is orthogonal to model size. Also, if you're using an agenti…

That's just... not true. Just compare any open model which is trained with the same recipe but multiple sizes.

You can compare models at OpenRouter site. Qwen 3.6 dense is in top 24% for coding.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#114

Post by otabdeveloper4 »

redox99 wrote: post #99
otabdeveloper4 wrote: post #97

Earlier quoted context omitted.

Number of parameters doesn't make the model smarter, it just makes it know more stuff out of the box. At some point there's diminishing returns and your coding LLM performs worse because you encoded useless stuff like Pokemon combinations or languages you don't speak into its parameter space. The "smartness" of the model comes from RLHF post-training, which is orthogonal to model size. Also, if you're using an agenti…

That's just... not true. Just compare any open model which is trained with the same recipe but multiple sizes.

> Just compare any open model which is trained with the same recipe but multiple sizes.

That's exactly what I did.

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#115

Post by NicoJuicy »

rirze wrote: post #14
NicoJuicy wrote: post #10

Earlier quoted context omitted.

Rtx 3090 24 gb set me back 390€ a year ago ( 2nd hand)

Was it still in good condition? That price makes me wonder if it was used for crypto mining, which can wear down the hardware.

It was from a belgian second hand market, probably they didn't knew what they had.

Works flawlessly

Re: RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

#116

Post by cybertim »

iMil wrote: post #46
cybertim wrote: post #44

Earlier quoted context omitted.

$ nvidia-smi topo -p2p r GPU0 GPU1 GPU0 X CNS GPU1 CNS X i guess not, i use llama.cpp with: --spec-draft-n-max 3 --spec-type draft-mtp --split-mode tensor --tensor-split 1,1 and my (gen) tk/s are between 60-80 tk/s will test this uncensored model and ngram added as well this weekend btw, i also set my powerlimit to 220watt per card (with nvidia-smi) that will cost you around 1 tk/s but safe you a LOT of power and hea…

CNS means Chipset not supported and I doubt it is the case, are you sure you are using the patched nvidia module? modinfo nvidia to check which one is loaded

Ok, i went through the P2P rabbithole, and incase anyone reads this, just use Fedora Rawhide, and run that repo install.sh mentioned in the blog post :) Now i have:

  GPU0 GPU1

 GPU0 X OK

 GPU1 OK X
Post reply on HN