Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Was my $48K GPU server worth it?

rosmine.ai

Post reply on HN
Print view View original

461–470 of 480 posts

Re: Was my $48K GPU server worth it?

#461

Post by digitaltrees »

rithdmc wrote: post #424
dpark wrote: post #415

Earlier quoted context omitted.

GP didn’t say they were defrauded. They said the listing was a cover for laundering money.

"weren't scammed" might have been a better choice of words, which they said one post up.

I said there are lots of scam listings. And then discussed a second issue which is fake listings that are designed to launder money. There are both. Some of the fake listings are obvious: way under priced, listings for “original boxes” not the actual computer. So that’s why I don’t want to buy off of eBay for this category. But I did also buy what appeared to be a reasonable listing and it was canceled. I can’t prove why but two sided money laundering is a known thing

Re: Was my $48K GPU server worth it?

#463

Post by ankit219 »

PowerElectronix wrote: post #298
miki123211 wrote: post #227

Earlier quoted context omitted.

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…

On top of that, AI providers are also eating a big loss on the service.

at a gross margin level, mostly no. if you include the cost of training a model as full R&D then possibly yes.

Batch size is what you should look at. If a cluster is running and processing one request, filling the batch has almost no marginal cost (kv cache creation/storage/fetch costs aside). But if the concurrent requests exceed batch size, one extra request would cost basically the rent cost of entire new cluster. APIs have the bursty nature so companies would plan to price it such that they are profitable / break even at 40%-50% utilization (% of filled batch for simplicity). so any extra request would not have the same cost as long as they are alongside an api request. you might think it degrades teh performance. easy: just assign a priority tier to api requests, and a lower tier to subscription requests.

its even more effective and powerful now that you have continuous batching. so likely if the api is being used, they are not eating any loss, let alone "big loss"

Re: Was my $48K GPU server worth it?

#464

Post by girvo »

adastra22 wrote: post #313
girvo wrote: post #291

Earlier quoted context omitted.

Yes I could, but that is annoying because of spot pricing and having my instance shut down, and it has fluctuating prices It’s also annoying because then I need to make sure my little “lab” setup is well automated, and I’m lazy :) Also, I literally said “ It's not financially a good idea” so I’m confused why you think I don’t know that.

Spot pricing and instance availability don’t apply to on metal hosting. You’d have your own machine dedicated to your own use only, at a locked in price.

Wait, I went and looked far and wide today and I can’t find anything with ~100GB of VRAM that isn’t $20k a year, what am I missing

Re: Was my $48K GPU server worth it?

#465

Post by pelasaco »

kube-system wrote: post #408
pelasaco wrote: post #278

Earlier quoted context omitted.

> A cage[0] is ~100x larger than what you need to host a single server. Yup, but i was assuming that he wanted to experiment building gpu rigs. For sure standard GPU servers are cheaper and easy to maintain. I have two lenovos, bought them used, already EOL.. was cheap and better than any custom gpu rig.. but i was pragmatic, because my goal was to put it in production, and not to research...

You could fit 10 million dollars of GPU rigs in the smallest of cages. A cage is an entire room. You don't need that to run a few servers.

> A cage is an entire room.

Not everywhere. Running a data center myself, we have cages for 1 rack, 3 racks and so on..

Re: Was my $48K GPU server worth it?

#466

Post by kappuchino »

freediddy wrote: post #10

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…

I have two of the M3s due testing of models at work and with exo I can run decent quantization with 1 millon tokens for memory and derailment tests.

Slow? Yes, but ... private. Unconditionally.

And recently with https://github.com/antirez/ds4 one can use just one system to a very, very decent speed and ttft for chat inference. Again, private.

Re: Was my $48K GPU server worth it?

#467

Post by BLKNSLVR »

jiggunjer wrote: post #217
manyatoms wrote: post #185

Earlier quoted context omitted.

"inflation"

"appreciation"

My phrasing was just to point out things moving in the opposite direction to what has previously been considered normal (which is also the economically intended 'normal').

Re: Was my $48K GPU server worth it?

#469

Post by miki123211 »

cootsnuck wrote: post #249
miki123211 wrote: post #227

Earlier quoted context omitted.

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…

Yea. LLM inference requires batch processing to have a shred of hope at being cost efficient. Batch processing requires a not so insignificant amount of scale (but probably not as much as people think). I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks. Trying to run bigger LLMs for an individual user to do big tasks is not…

I think people overstate the importance of batch processing and understate the importance of consistent workloads.

A GPU is much like a hotel or an airplane. To make hotel and airplane economics work out, you need to keep your seats / rooms filled at all times. Because the number of spots is limited, an empty seat, room or request slot today is an opportunity forever wasted.

Unlike hotels and airplanes, though, GPU requests are short-lived. This means you need somebody to provide you with useful work to do 24/7. If you're an individual who needs to sleep or a single-country business that works 9 to 5, most of your request processing opportunities will go to waste. You're much better off renting your GPU from somebody who can provide it with useful work 24/7, and hence get much better per-request prices.

Re: Was my $48K GPU server worth it?

#470

Post by plutokras »

ttoinou wrote: post #149
mountainriver wrote: post #115

Earlier quoted context omitted.

Running LLMs on Macs is still terribly slow. They simply lack the optimizations other platforms have. An RTX 6000 pro Blackwell is a pretty good card

M5 pro 48GB should be good and future proof

Last time I looked into it, I realized how severely it's limited by memory bandwidth. Only the M5 Ultra compares to a dedicated graphics card, but it still falls short.
Post reply on HN