Earlier quoted context omitted.
Not even a single mention of gaming. No wonder gamers hate AI bros.
I have a second computer with an RTX 4090 for gaming (running Windows). I also used the new RTX 5090 running Linux to evaluate whether Proton/Wine allow me to run Windows games on linux (yes, it works, but the compatibility and frame rate issues make me stick to native Windows for now).
Was my $48K GPU server worth it?
221–230 of 480 posts
Re: Was my $48K GPU server worth it?
#222Sounds fun/stressful/rewarding. I'm most interested in the update at the end though 'Launch was a success! 400K+ views, and multiple companies reached to use my IP.' I too, like probably 1 in 5 of the people reading this, think I have figured out some major problems with LLMs (context and computation research) but have wondered the best way to 'release' and get value out of it. I can see training being a little easie…
My first advice is to have a test set with clear improvement, and a clear "wow" demo use case. There are lots of "breakthroughs" that seem good but aren't (e.g. some new architecture that doesn't mask past tokens correctly and leaks information), so people will assume it is wrong. To prevent this, you need to be extremely rigorous in your launch materials. If you can make it into a product that people can try out themselves, that goes a long way. You don't need to open source any code (I haven't yet) if people can try it out some other way like a demo website. Good luck! Ping me if you want to chat more
Re: Was my $48K GPU server worth it?
#223Earlier quoted context omitted.
I’m not usually one to ask this because learning to do a thing can be fun, but why exactly have you spent 25 thousand dollars on getting an LLM someone else made to answer maths exam questions?
I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down. It's not financially a good idea: renting reall…
$6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)
Re: Was my $48K GPU server worth it?
#224I can't imagine spending $48K on a home GPU server, but I did just splurge and buy a PC with an RTX 5090, specifically to hold the largest models you can fit in 32GB. It's a top of the line PC with water cooled high end CPUs, 64GB RAM, RTX 5090 for $5K. To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade. I don't run it at 100% power (it's mostly idle…
I was looking at Ultras for sale, and had same worry, so didn't end up getting one. I have some peace of mind comfort about applecare and technical repair, but i couldn't find insurance that would cover theft (or rather, i did, but it was too expensive)
Re: Was my $48K GPU server worth it?
#225This is a difficult calculation to make because you wouldn't rent time on the exact same system in the cloud. Depending on what you're running, a bigger server with better inter-GPU interconnects in the cloud might complete the task so much faster that the additional per-hour expense is more than covered.
Right, you can rent from a v100 from llama cloud for $0.79/hr. An h100 is $3.99 /hr.
$48000 is equal to 12000 hours of renting an h100, which is about as long as you’d spend at your job for 6 years!
Re: Was my $48K GPU server worth it?
#226Earlier quoted context omitted.
Nitpicking, but the worst case of spending $25k is unforeseen circumstances that write off the entire asset. I don’t think -$2000 is a conservative enough figure for standard depreciation either (a lot can happen in a year)
Either I don't understand the used apple market.. or I agree this is crazy. Someone spends $25k on new hardware, waits a year, and expects to sell it for $23k? Unless the ram issues save him, and cost of new goes up, I don't see how that was going to work.
Apple products have had relatively high resale for a while. Only losing 8% in a year is probably extra unusual, and 1-year-old wasn't really ever the sweet spot, but a "sell used privately after a few years, roll onto the new one" has been a relatively common play.
Doing this particular one is definitely expecting the market squeeze to continue. "Worst case" is back to more "normal" depreciation. Where I'd expect to only be able to recoup more like 18k. But... if you look at GPU prices the last 3 years... it's not a crazy assumption that it won't drop that fast.
iPhone example since those are easiest to find in quantity: new iPhone 16 Pro Max for $1200, Gazelle would want $866 for "execllent" condition. Lost ~28% for one-model-back. iPhone 15 Pro Max, though: excellent priced at $667 here, only down another 23%, and gives you basically half-priced-upgrade if you can sell it for that and roll into the newest.
So to have never-more-than-one-model-old rough estimate at today's value-holding you'd be out $3600 for three new phones, with getting 1732 of that back, or 1868 for it (with a $334-per-year incremental cost of upgrade).
For never-more-than-two-models-back you'd be out $2400, getting back $866, for net $1534 spend, with a $167 incremental per-year upgrade cost once you buy the first one. Pretty good if you keep the phone in excellent condition and are happy to budget a bit over $10/month to be on a every-two-year upgrade train.
Well, you'd also eat the tax...
https://buy.gazelle.com/products/iphone-16-pro-max-256gb-unl...
https://buy.gazelle.com/products/iphone-15-pro-max-256gb-unl...
Re: Was my $48K GPU server worth it?
#227In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…
This is, sadly, obvious and inevitable in retrospect.
The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself.
This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.
Re: Was my $48K GPU server worth it?
#228Earlier quoted context omitted.
I got an RTX 6000 pro too. I like running locally, I've learned a lot more than if I had used an API and there's less worry about overspending tokens. I accidentally spent $100 on claude api in like 2 days because I didn't know what I was doing. The problem is that while one these gpus is a huge improvement over a laptop or a single 3090, you very quickly wish you had more. I would buy a second one, but I did the mat…
A pair of RTX6000 cards will give you a good performance boost due to tensor parallelism, though. I haven't tried the newest predictive quants but I see about 35 tps when running the 8-bit Qwen 3.6 27B model on one board and about 50 tps on two. Probably could come close to 100 tps on an optimized setup with the latest GGUFs. Also, the 4-bit quants of MiniMax 2.7 will run at 100 tps or so with two cards, which is pre…
You can get 70-80 tps on qwen3.6-27b f16 with MTP on a single card
Re: Was my $48K GPU server worth it?
#229Earlier quoted context omitted.
Either I don't understand the used apple market.. or I agree this is crazy. Someone spends $25k on new hardware, waits a year, and expects to sell it for $23k? Unless the ram issues save him, and cost of new goes up, I don't see how that was going to work.
Well, Apple is literally not offering the M3 512GB studios currently. You can’t even back order one. They are selling on EBay for over $20k, used.
It’s hard to know if any of these eBay listing are real or actual sales. Lots of scams.
Re: Was my $48K GPU server worth it?
#230I'll buy it from you!