461–470 of 1001 posts
I find this analysis confusing. PMF for coding was likely reached some time last year. Profitability, which is different, we don’t know. The article kind of confuses both without making a strong economic case or using numbers in a compelling way. I don’t understand what the Uber case has to do with this either. The Uber COO clearly said that at least in terms of ROI he’s not seeing the results either. My take is the…
"I’ve called November 2025 the November inflection point because that was when GPT-5.1 and Opus 4.5, combined with their respective coding agent harnesses, got good—good enough that we’ve spent the last six months adapting to agent systems that can reliably get useful work done."
Earlier quoted context omitted.
Inference has traditionally been far less expensive than training. One public example is the fact that hobbyists can run StableDiffusion ($600k training costs[1]) on their personal computers. Speaking to your point, inference being dramatically less costly than training would not be seen as a delta from the norm. The model of providing inference for anything near the operational costs (like a utility would), would th…
The difference between training and inference is 1) one have to keep intermediate results for backward pass in training and 2) computation for training double because of the backward pass. Training is also done over batches, which increase memory requirements by several orders of magnitude. This is why training needs costly compute. One of the ways out of this unfortunate situation is to use something like Stochastic…
Training is inference + backwards pass (~2x inference cost) + activations (vram overhead) + optimizer (vram overhead) + gradients (vram overhead).
Earlier quoted context omitted.
> ...we are already looking at dropping $100k on hardware to run local models... Just think how much further that $100K would have gone if the hardware market wasn't so screwed-up. Anecdote: I priced-out adding 1TB of RAM to a four node cluster a couple months ago. The cluster was purchased in fall of 2024 w/ 4 nodes, each with 256GB RAM. The nodes cost just over $14K apiece back in 2024 (entire box, not just the RAM…
> Dell wanted >$90K a couple months ago to add 256GB to each node. RAM is expensive, but not THAT expensive. I just bought 128Gb for about $5k for our build cluster (it's not even for AI, sigh). Even if you need larger-sized DIMM sticks, it's still going to be in the vicinity of ~15k tops.
I haven't had problems w/ Dell support and 3rd party memory, personally, but given the machines' application I understood the concern.
They've got, ballpark, $5t to $10t to make back in the next 5 years, or the hardware buildouts will start getting written down. This means we're going to need $1t+ per year in spending, per year, on tokens. 200m knowledge workers in the world, 30m developers. We're talking about a world where you need 5% of every knowledge workers salary to go into tokens. 20% if you're a developer. That's a _huge_ shift. Most people…
And that's not considering that capitalism is going to do what it does best: if they really found a way to be profitable, competitors are going to fight them on pricing. Anthropic, OpenAI, Google, etcetera 's margins are a competitors' opportunities.
It's not as if there weren't chinese models nearly SOTA. Don't know where the french (Mistral) are but they may try to get in the game if there's a way to be profitable (not that France or the EU for that matter are relevant in anything tech or had any tech company besides ASML and SAP in the Top 100 but who knows).
They've got, ballpark, $5t to $10t to make back in the next 5 years, or the hardware buildouts will start getting written down. This means we're going to need $1t+ per year in spending, per year, on tokens. 200m knowledge workers in the world, 30m developers. We're talking about a world where you need 5% of every knowledge workers salary to go into tokens. 20% if you're a developer. That's a _huge_ shift. Most people…
What I'm often hearing though is the equivalent of "gg ez" when I bring that up. I don't understand how this will at any point blitz scale to profitability. As far as I know they don't have positive cash flow, no one has a moat and I don't think they will push out engineers.
> $2,180.16 worth of tokens for $200 “Tokens” don’t have an intrisic cost or value. Saying that I used $2,180.16 worth of tokens is like relying on the salesperson to convince me I’m getting a billion dollars worth of pots and pans for $19.99. I think it’s funny how we are throwing critical thinking out the window when it comes to evaluating biased sources of info.
I'm not sure what you're pushing back against here. I spent $200. If I had been paying API pricing it would have been $2,180.16. The article is about how enterprise customers get charged API pricing, which means if I had been employed by one of those companies I would have cost them $2,180.16. What am I missing?
Does that mean you'll be saving $99k?
It sounds an awful lot like the mark-up to mark-down scheme where the price stays the same.
Earlier quoted context omitted.
I work for a tiny little company ($150MM annual rev with 9% net) and we are already looking at dropping $100k on hardware to run local models because, for us, they're "good enough." Our estimated spend for AIaaS would exceed that cost in less than a year. In a few years, there will be hardware capable of running frontier models good enough for most things at accessible prices for even tiny companies.
Yeah, that's the part that just seems to be wildly under-discussed to me. If open source models are ~3-6 months behind SOTA, and ~opus4.6 capabilities are good-enough for product market fit, do the frontier labs have half a decade to catch up on their prior burn? AI cost ballooning faster than companies can afford is becoming a very common topic in my circles right now. The era of "I'll pay infinitely more for margin…