21–30 of 128 posts
Garbage. You can't include training by the companies that develop an llm in the comparison against companies that merely use the same llm. Apples and potatoes.
OpenAI and Anthropic aren't charities, so whatever cost they inccur for training will be passed down to the companies using the models. So you absolute should include it.
We don't get unlimited hiring budget, so we also won't get unlimited token budgets, and we as the operators will be responsible for the productivity of our agents.
What does performance management for engineers look like when dollar token cost is included in reviews? I think it's going to change a lot of assumptions and a lot of strategy around AI use.
> The rest of the software market trails.
This shows how VC firms see things and why we have such a lopsided market where grift rises to top easily.
Yes the rest of the software market trails in comparision to the compute costs at Anthropic if you including training the actual models. Like is this the insight? Biggest AI company spends a lot of money to make AI models?
Sure you can find anthropic's business model risky/not feseable but using this as your starting point shows a lack of basic understanding at best and malicious intent to make a stupid point at worst
Earlier quoted context omitted.
I don't know, compute is compute. Arguably making complex software with LLMs isn't all that different from training a model to do a thing. You're throwing a lot of compute at the problem and hoping for a stochastic solution. The distinction will become even blurrier with time. Though I agree it might be informative to split it by industry sector.
If you’re going to include AI training in costs, you should include education as part of the costs of an engineer …
Earlier quoted context omitted.
Exactly, it's like saying Shell is spending a fortune on fuel compared to what they spend on employees, if you count oil extraction costs as 'fuel'.
So where are these training costs getting paid from?
Big AI labs are not software companies where payroll dominates expenses. They're capex-heavy industrial entities; it just so happens that the "machines" (whose output they sell) are nominally the same category as the devices that their knowledge worker employees use on their desks.
Ignoring the bizarre inclusion of training compute for the AI company estimates, the other comparisons are still valid. > The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI, 40% of a fully-loaded $224k senior engineer salary. The median spends $137. That is the gap : ... 0.4x at the top of the market, near zero at the median. So it's not more expensive than an engin…
Does it though? I do not see any advantages in my day to day job over using the cheaper models.
So, overall, you get more done that without AI, at the cost of spending almost all of your time writing specs and doing code review and almost none of it writing code.
Do you get 3.3x the work done? Probably not. Do you get 2x the work done? I think maybe, if you can hack the dynamics of the new job as a manager of eager robots. For me the jury's still out on the second point.
Open-weight models are going to completely shatter these forecasts. It takes a little more effort – right now, probably won’t be true in three months – but you can achieve the same at 1/10th of the cost.
For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.
Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.