281–290 of 480 posts
The other advantage of the local GPU is that you are not feeding your data into cloud providers. I'm not sure how much you can really trust Anthropic and OpenAI not be improving their models based on your input.
It doesn't cover risk. If one or more gpus dies, who pays for it? If you rent, you are guaranteed to be insulated from this risk. But owning, you might not have the best return policy from the vendor. And if you are actually at fault for breaking it, they have every right to deny a return. Or if your apartment is burglarized or catches fire (possibly from overloading the circuit) you are out the entire investment.
Earlier quoted context omitted.
Privacy and offline operation are valuable or non-negotiable in some cases, but the difference is pretty categorical between what can run on a single card and what can run on a DGX GB200 NVL72 cabinet. Doesn't mean it's not worth seeing how far local models can be pushed. Not every problem needs a senior engineer.
I know it's one of those "if you have to ask" situations, but curiosity got the better part of me. Here's the search assist response: "The DGX GB200 NVL72 AI server costs approximately $3 million per unit. This system includes 72 Blackwell GPUs and 36 Grace CPUs, making it one of the most powerful AI servers available." The search assist actually credited a source used with: https://www.tweaktown.com/news/98292/nvidi…
There are bigger data centers than Colossus 1 around too.
There is a reason NVidia is the most valuable company on the planet.
https://en.wikipedia.org/wiki/Colossus_(supercomputer)#Curre...
I am not even going to pretend that this is financially reasonable option. I simply wanted to have a local models. Maybe down the line, as cloud models become less subsidized, I might benefit from having a local setup, but for now, it wasn't the most prudent financial decision.
But one big benefit is that I never have worry about my account being randomly banned nor I have to worry about running out of quota. I still use codex and opus for some specific tasks, but as tools are improving, I need them less and less.
Earlier quoted context omitted.
I'm not entirely sure what your point is here; me asking for humans to give an opinion does not preclude me from also asking AI.
I was just making a correction based on what you said. "AI is cool but it's not going to have all the good and bad experiences that humans have had with different motherboards." AI will have more access to experiences than you'll find here.
I feel like there is some very deep generalizable wisdom buried here.
In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I'v…
This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…
Earlier quoted context omitted.
This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…
Historically it was not uncommon for beds to be rented out to multiple people.
Earlier quoted context omitted.
This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, a…
Historically it was not uncommon for beds to be rented out to multiple people.
Earlier quoted context omitted.
Either I don't understand the used apple market.. or I agree this is crazy. Someone spends $25k on new hardware, waits a year, and expects to sell it for $23k? Unless the ram issues save him, and cost of new goes up, I don't see how that was going to work.
Apple products have had relatively high resale for a while. Only losing 8% in a year is probably extra unusual, and 1-year-old wasn't really ever the sweet spot, but a "sell used privately after a few years, roll onto the new one" has been a relatively common play. Doing this particular one is definitely expecting the market squeeze to continue. "Worst case" is back to more "normal" depreciation. Where I'd expect to…