Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I put a datacenter GPU in my gaming PC

blog.tymscar.com

Post reply on HN
Print view View original

31–40 of 199 posts

Re: I put a datacenter GPU in my gaming PC

#31

Post by Aurornis »

mickeyp wrote: post #25

Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat. It's prefill; slow prefill kills agentic workloads dead. If you have 100,000 tokens at ~150tok/s per the OP, you're looking at: You have: 100000 / (150/s) You want: hms 11 min + 6.6666667 sec Which is quite a wait indeed.

Most people won’t be dumping 100K tokens into it at once, but I agree that all of the prefill time that adds up during a session becomes a lot to account for.

This is also a problem for all of the Mac local LLMs. Macs are a great way to get a lot of high bandwidth memory, but their compute is very far behind current gen dedicated GPUs. Some of the expensive Mac Studio setups allow you to run very large models with usable tokens/s, but you can be waiting a long time for it to get to the point of generating those tokens.

Re: I put a datacenter GPU in my gaming PC

#32

Post by tgv »

postalrat wrote: post #8
lelanthran wrote: post #3

> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Because humans write exactly like this /s

Where do you think llms learned to write that way?

Because their custom training data contains an emphasis on such verbiage. It doesn't come from the God-knows-how-many TB of web content the model is pre-trained on. There, such phrasing is only a drop in the sea. But the "yes, you're right" phrases, the em dash, etc., come from the later stage, for which content is created according to some (probably overprecise) guidelines.

Re: I put a datacenter GPU in my gaming PC

#33

Post by qingcharles »

peddling-brink wrote: post #29
m0rde wrote: post #24

Earlier quoted context omitted.

What a difficult world you must live in these days

While I don’t disagree with their sentiment, I’m far more annoyed with it than the AI writing.

Every single HN post has the same comment now.

Re: I put a datacenter GPU in my gaming PC

#34

Post by fouc »

recursivegirth wrote: post #19

> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Had to stop there. Annoying. I can't stand AI use for writing. It makes any otherwise great article feel so disingenuous.

That line was the exact moment I also realized the post was AI written. I kept reading though, but I am left constantly guessing at which key details might be pure hallucinations.

Re: I put a datacenter GPU in my gaming PC

#35

Post by rafram »

qingcharles wrote: post #33
peddling-brink wrote: post #29

Earlier quoted context omitted.

While I don’t disagree with their sentiment, I’m far more annoyed with it than the AI writing.

Every single HN post has the same comment now.

Only because so many of the articles posted on HN now are AI-written, and badly, too. A lot of tech people are so impressed with LLMs’ capabilities in code that they fail to recognize how bad they are at writing enjoyable prose. And it feels like a chore to write out a whole blog post by hand when the machine could do it for you! But the result we get is so, so much worse and more annoying.

Re: I put a datacenter GPU in my gaming PC

#37

Post by wookmaster »

Alifatisk wrote: post #22
casey2 wrote: post #9

Some resell group is going to have to make this easier. The shear amount of these cards otherwise heading towards the landfill is staggering. That is if Big Tech don't destroy them to prevent model weights from leaking.

> The shear amount of these cards otherwise heading towards the landfill is staggering. The thought of throwing away working cards sounds so bizarre to me. I can't believe companies would dispose them into the landfill like that, it is at least worth giving away for refuse.

There’s a long history of corporations doing evil things to ensure their business model succeeds

Re: I put a datacenter GPU in my gaming PC

#38

Post by hattmall »

lelanthran wrote: post #28
postalrat wrote: post #8

Earlier quoted context omitted.

Where do you think llms learned to write that way?

> Where do you think llms learned to write that way? Not from individual human content, that's for sure - maybe MLM marketing copy? Sleazy 4AM ads? I mean, every time this response comes up, I keep asking the person to point at something written prior to 2022 that gets 80%+ on the LLM detectors, and yet no one can find anything. Maybe you, postalrat, can find something written in this style that was published prior t…

It's a function of the LLM "thought process"! It's not really modeled after human speech. It is in short segments but not long form, same reason you see the same rather odd nuances in LLM generated code.

If they way you thought was to run a bunch of if statements, generate content, then feed that content back to get a "score" of what seems the most plausible, run the if statements again, and adjust / merge responses, then you would write similarly. The recognizable cadence of LLM generated content is pretty clearly the result of a lot of if statements being fused together.

Re: I put a datacenter GPU in my gaming PC

#39

Post by bob1029 »

> And yes, if you want the absolute best, Opus 4.8 exists. It also costs more per 20 minutes of heavy use than I paid for this entire GPU and adapter setup combined. But the gap is shockingly small.

I don't think this is a fair characterization of the situation. I use frontier models via API pre-paid tokens every single day, and I can barely rack up $100 per month. The fact that we figured out how to burn double this in 20 minutes is impressive, but I don't think it reflects the reality that many are experiencing right now. There are some exceptionally gluttonous approaches to harnessing LLMs that I think are serving as convenient straw men in these discussions.

Paying for the API will almost always be more economical than self-hosting equivalent infrastructure. I am not against self-hosting, but the article suggests a primarily economic motivation for this effort. If you are consuming fewer than 10^9 tokens per month, I really don't think it's worth your time to try and compete with the hyperscalars. Most of the money is to be found in the integration of this technology with existing businesses.

Re: I put a datacenter GPU in my gaming PC

#40

Post by bitwize »

lelanthran wrote: post #3

> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising. Because humans write exactly like this /s

You know what the sad bit is? Humans do write exactly like that. That's not even particularly egregious StalkedIn marketroid speak.
Post reply on HN