Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I put a datacenter GPU in my gaming PC

blog.tymscar.com

Post reply on HN
Print view View original

141–150 of 199 posts

Re: I put a datacenter GPU in my gaming PC

#141

Post by lelanthran »

tymscar wrote: post #128
lelanthran wrote: post #126

Earlier quoted context omitted.

GPTzero says 100% AI generated for specific paragraphs that I chose (such as `Multi-token prediction`). If you remove all the code listings, tables, etc and just paste the prose into these tools, it drops to 87% AI generated. None of the 3x older blogs of yours that I tried went above 5% AI generated. Maybe you're spending so much of time with the LLM that you are talking like it; in which case, take an old blog and…

GPTzero is a joke. It thinks this is AI: “I bought a datacenter GPU that doesn’t even have a normal PCIe connector, stuck it in my gaming PC with an adapter, and now I have 32GB of VRAM across two GPUs running a 27 billion parameter model at 32 tokens per second.” There’s nothing AI about that. Not all SOTA LLMs agree, hell, none of them do. The same exact example I sent here gives me 0% in some, 10% in others, 100%…

> Not all SOTA LLMs agree, hell, none of them do.

The ones I checked all agree: your recent writing is not the same author as your writing from 3 years ago...

You can check this yourself if you don't believe; make of that, what you will.

Re: I put a datacenter GPU in my gaming PC

#142

Post by kethinov »

sonzohan wrote: post #100

I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article: Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models. Nitpick,…

qwen3-coder-next runs fine on my consumer grade nvidia 4070. Performance is not spectacular, but it's only a little bit slower than a properly-fit model.

Re: I put a datacenter GPU in my gaming PC

#143

Post by solenoid0937 »

vidarh wrote: post #58
bob1029 wrote: post #39

> And yes, if you want the absolute best, Opus 4.8 exists. It also costs more per 20 minutes of heavy use than I paid for this entire GPU and adapter setup combined. But the gap is shockingly small. I don't think this is a fair characterization of the situation. I use frontier models via API pre-paid tokens every single day, and I can barely rack up $100 per month . The fact that we figured out how to burn double thi…

I use hosted providers myself, but I can churn through $100 worth of tokens in half a day even with cheap models like Deepseek easily. If someone's use is as light as yours, then sure - grab a subscription and you'll save far more. For higher use it will come down to how cheap your electricity is whether it is worth offloading at least some of it (for me it's not, FWIW)

Same, very surprised when people on HN are shocked by high token burn - it's really not hard if you've figured out how to use LLMs!

Re: I put a datacenter GPU in my gaming PC

#144

Post by sonzohan »

kethinov wrote: post #142
sonzohan wrote: post #100

I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article: Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models. Nitpick,…

qwen3-coder-next runs fine on my consumer grade nvidia 4070. Performance is not spectacular, but it's only a little bit slower than a properly-fit model.

What are your settings and tokens/second? Even with 2 GPUs (MI100, RX 6600 XT 8GB) and 32GB of RAM it was running at a snails pace for me.

I didn't try a sched_spread with a 3090 and the MI100 which would provide 56GB ram

Re: I put a datacenter GPU in my gaming PC

#145

Post by rft »

sonzohan wrote: post #136
Silagi wrote: post #120

Earlier quoted context omitted.

Did you consider the R9700 or B70 when you went for the MI100? If so, what made you choose the MI100? I've been playing with picking up a card in this class but haven't been able to justify it when running the Qwen3.6 MOE model on a 6800xt is tolerable for the type of projects I've been willing to point local AI at.

I looked at those, the Arc 1100, the w6800, MI50, MI60, v100, v620, and basically anything with 32gb of RAM: 1. I wanted an AMD card. 2. I have an RTX 3090 that's been fun to play with, but I want to get back to using it for gaming. 3. I was looking for between 30-60 tokens/second in terms of performance on the beefier models I want to run. Looking at stock Qwen3 32B the benchmarks reported about 41 tokens/second for…

> The MI100 is roughly double the performance on Qwen 3.5 35B A3B Q5_K_M to the R9700 (462 token/s prefill vs 239 tokens/s, 217 tokens/s vs 118 token/s for inference)

Those prefill numbers look really low to me. I can run nearly that same model (qwen 3.6) at q4km with q6 cache on a single 3090 and get 2.3k-4.4k prefill and 100-170 generation. Just based on raw numbers I would expect the R9700 to land around 70-90 generation (about 2/3 of memory bandwidth of a 3090) and at least the same or higher prefill (nearly 3x FP16 TOPS on the R9700). That means the numbers really don't add up. Is the benchmark done with some special settings, e.g. parallel requests or with very low prompt length?

Re: I put a datacenter GPU in my gaming PC

#146

Post by codebolt »

vidarh wrote: post #92
iJohnDoe wrote: post #83

Earlier quoted context omitted.

Could you share a bit about what you’re working on or what type of projects require that much usage? Is it hobby, production, revenue generating?

A mix. I have hobby projects that churn through that much when I don't need the tokens for others things. I also have projects for clients that easily consumes those levels. As well as a stealth-ish potential startup. Currently I'm at 4 different subscriptions + more than I'd like in spend via OpenRouter... What multiplies it very quickly is when you start feeding them with test suites and "Ralph loops" that run unti…

I'm skeptical of letting agents run free like this. Even Opus makes decisions I don't always agree with. And I quickly lose my mental model of how the code is evolving.

I get more enjoyment and better results when the coding process is me and the agent working through a plan, at each step sparring over what to do next and how. Then I also catch the bad decisions before they manifest in the code.

Re: I put a datacenter GPU in my gaming PC

#147

Post by pyrolistical »

Aurornis wrote: post #31
mickeyp wrote: post #25

Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.

Most people won’t be dumping 100K tokens into it at once, but I agree that all of the prefill time that adds up during a session becomes a lot to account for. This is also a problem for all of the Mac local LLMs. Macs are a great way to get a lot of high bandwidth memory, but their compute is very far behind current gen dedicated GPUs. Some of the expensive Mac Studio setups allow you to run very large models with us…

The prefix cache is working properly 100k doesn’t prefill more than once

Re: I put a datacenter GPU in my gaming PC

#148

Post by overfeed »

sonzohan wrote: post #100

I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article: Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models. Nitpick,…

> You will need to buy, at least, a bunch of decent 120mm fans to prevent this or invest in some water cooling

There's a cottage industry of 3D-printed fan-shrouds for data center GPUs - 120mm are often the sweet spot for quietness and practicality. The shoud smugly fits the GPUs intake, so it gets all the airflow from the attached fan(s), whose speed curves can be attached to GPU temperature.

Re: I put a datacenter GPU in my gaming PC

#149

Post by sonzohan »

rft wrote: post #145
sonzohan wrote: post #136

Earlier quoted context omitted.

I looked at those, the Arc 1100, the w6800, MI50, MI60, v100, v620, and basically anything with 32gb of RAM: 1. I wanted an AMD card. 2. I have an RTX 3090 that's been fun to play with, but I want to get back to using it for gaming. 3. I was looking for between 30-60 tokens/second in terms of performance on the beefier models I want to run. Looking at stock Qwen3 32B the benchmarks reported about 41 tokens/second for…

> The MI100 is roughly double the performance on Qwen 3.5 35B A3B Q5_K_M to the R9700 (462 token/s prefill vs 239 tokens/s, 217 tokens/s vs 118 token/s for inference) Those prefill numbers look really low to me. I can run nearly that same model (qwen 3.6) at q4km with q6 cache on a single 3090 and get 2.3k-4.4k prefill and 100-170 generation. Just based on raw numbers I would expect the R9700 to land around 70-90 gen…

Numbers are from https://www.fitmyllm.com/ so they're not a real hardware benchmark just what you're expected to get. YMMV.

Re: I put a datacenter GPU in my gaming PC

#150

Post by rft »

sonzohan wrote: post #149
rft wrote: post #145

Earlier quoted context omitted.

> The MI100 is roughly double the performance on Qwen 3.5 35B A3B Q5_K_M to the R9700 (462 token/s prefill vs 239 tokens/s, 217 tokens/s vs 118 token/s for inference) Those prefill numbers look really low to me. I can run nearly that same model (qwen 3.6) at q4km with q6 cache on a single 3090 and get 2.3k-4.4k prefill and 100-170 generation. Just based on raw numbers I would expect the R9700 to land around 70-90 gen…

Numbers are from https://www.fitmyllm.com/ so they're not a real hardware benchmark just what you're expected to get. YMMV.

Ah, ok. I took a look at the 3090 numbers and they list 400 tok/s prefill, so if I normalize my expectations to that base line the numbers you posted do make sense. I haven't dug deep into that site's methodology, but their estimates seems way off. Especially since they don't take into account cache quant when deciding whether or not you can run a model. Overall I found that website a bit confusing, but maybe the UX just didn't click with me.
Post reply on HN