Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

431–440 of 480 posts

Re: Was my $48K GPU server worth it?

#431

Post by nine_k »

59nadir wrote: post #309
gpt5 wrote: post #268

Earlier quoted context omitted.

Sell the machine for $4K, use it to pay for Codex Pro for everyone for a year. Everyone will be significantly more productive and happy. It's not even a real comparison if they are actually using them for coding. If you are deploying always running agents (e.g. monitoring logs and services) then sure - a QWEN local server is a good choice. But for coding the cost in productivity of using a lower performing model is w…

Anyone who frivolously suggests throwing away possible independence in favor of dependence on a Silicon Valley company is either incredibly naïve or acting in bad faith.

Not necessarily so. I can see how a bid to predict how thing will be in 1 year in AI-based coding is likely a losing one. So the idea is to extract the maximum value now, and turn it into profits that would buy you whatever is adequate for the next steps. For comparison, the AI-based coding landscape a year ago, in May 2025, wasn't even close to what we have now, and half the key tools did not exist.

OTOH, as we see, the larger models demonstrate diminishing returns, smaller models demonstrate improvements, and hardware does not show any signs of becoming cheaper, so holding on existing decent GPUs may, too, be a winning strategy in longer term.

Re: Was my $48K GPU server worth it?

#432

Post by [deleted] »

throwawaytea wrote: post #265
tombert wrote: post #216

Earlier quoted context omitted.

I'm not entirely sure what your point is here; me asking for humans to give an opinion does not preclude me from also asking AI.

I was just making a correction based on what you said. "AI is cool but it's not going to have all the good and bad experiences that humans have had with different motherboards." AI will have more access to experiences than you'll find here.

[deleted]

Re: Was my $48K GPU server worth it?

#433

Post by p1esk »

r0b05 wrote: post #344
kgeist wrote: post #335

Earlier quoted context omitted.

They don't use the server all at once. In the UI, users typically ask a question, get a response, and continue with their work. In the case of autonomous agentic loops, an agent simply waits its turn until the server is ready to accept the request. Agents don't hammer the server 24/7 every second either, because they either need to be triggered or are busy doing other work, such as compiling or running tests.

It would be more interesting to know how many simultaneous users this setup can serve. Otherwise I can just say it serves 500 users but not all of them use it at the same time which doesn't communicate the right level of detail.

Depends on TTFT and tokens per second you want.

Re: Was my $48K GPU server worth it?

#434

Post by tom_alexander »

timothyduong wrote: post #164
datadrivenangel wrote: post #35

I did the math at least on a Macbook pro, and for inference it's definitely not worth it. - https://www.williamangel.net/blog/2026/05/17/offline-llm-ene... - Discussion: https://news.ycombinator.com/item?id=48168198

That's the case with Self-hosting anything. It is the privacy that matters.

Not necessarily. I was spending ~$150/month on vultr's kubernetes hosting. I spent $5k building out a pretty awesome 1U server and I put it in a colo that costs me $50/month. Next year I will break even financially and everything after that is saving money. I also am getting so much more out of this server than I was getting on vultr because I over-spec'd the machine. In addition to running more on my cluster, I spin up large virtual machines for development, experiments, and for offloading distributed builds. No shade to vultr, but owning my hardware instead of renting was absolutely the way to go. Unfortunately today the ram alone would cost over $5k, so the math has changed.

Re: Was my $48K GPU server worth it?

#435

Post by brightball »

krupan wrote: post #416
brightball wrote: post #404

Earlier quoted context omitted.

To some degree I think there's a hope that it becomes like a gym membership. If everybody used their membership, the gym would be too crowded. It's all of those memberships that people feel like they need to have but don't use where the extra profit comes in. As long as the power users are paying per token, everything is good.

Really? This is what we expect from this amazing world changing technology? People will sign up for it and not use it? Good business plan, how can I invest? /s

Just speculating on the math.

Re: Was my $48K GPU server worth it?

#436

Post by HWR_14 »

agentifysh wrote: post #281
HWR_14 wrote: post #246

The other advantage of the local GPU is that you are not feeding your data into cloud providers. I'm not sure how much you can really trust Anthropic and OpenAI not be improving their models based on your input.

You can turn off training with codex an gemini. Not sure about Anthropic.

I can check a box.

How much do you trust OpenAi or Anthropic to not use it as training data anyway? What if you are building a startup and they can just use their visibility into it to copy your IP instead of buying your company?

Re: Was my $48K GPU server worth it?

#437

Post by mixmastamyk »

542354234235 wrote: post #407
mixmastamyk wrote: post #239

Earlier quoted context omitted.

Are we talking about a cash transaction? If so >$10k is dangerous as the police may want to steal it themselves. If it is an electronic payment, I'm not sure how completing the transaction in front of a police station will help any. Well, it will help the buyer to see it working, but the seller gets no additional protection besides seeing "a person."

I love that it never occurred to you that the "buyer" could just steal the item. Be safe out there.

Ask a question, get a condescending answer.

Re: Was my $48K GPU server worth it?

#438

Post by iamtheworstdev »

kgeist wrote: post #150

I administer a simple AI server in the office, which just uses a single RTX 5090 but is able to serve ~80 people throughout the day. I'm impressed by Qwen3.6-27b's capabilities in agentic coding/tasks so far. Devs say it's not much different from Sonnet 4.6 on many tasks (sometimes it even outperformed it), 40-60 tok/sec, up to 260k context. The server cost about $10k with all the bells and whistles. I spent a lot of…

I thought NVLINK didn't matter anymore because of the latest PCI-E speeds. Am I wrong there?

Re: Was my $48K GPU server worth it?

#439

Post by MichaelZuo »

cootsnuck wrote: post #249
miki123211 wrote: post #227

Earlier quoted context omitted.

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…

Yea. LLM inference requires batch processing to have a shred of hope at being cost efficient. Batch processing requires a not so insignificant amount of scale (but probably not as much as people think). I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks. Trying to run bigger LLMs for an individual user to do big tasks is not…

Wasnt this pretty evident to pretty much anyone who knew even a bit about inferencing?

Idk what people were thinking. I’ve never seen anyone offer a plausible way to sidestep batch processing for example.

Re: Was my $48K GPU server worth it?

#440

Post by bigyabai »

heisenbit wrote: post #300
datadrivenangel wrote: post #35

I did the math at least on a Macbook pro, and for inference it's definitely not worth it. - https://www.williamangel.net/blog/2026/05/17/offline-llm-ene... - Discussion: https://news.ycombinator.com/item?id=48168198

One value of learning on my Macbook is that mps is not as well supported as cuda which forces me to go down roads I would not have traveled.

That's more of a disadvantage. CUDA is an industry standard, MPS/MLX/Metal compute shaders are a novelty.
Post reply on HN