111–120 of 162 posts
measure outcomes (impact), not effort (token usage, lines of code, code coverage, hours worked, etc.)
How?
This is an org pushing thousands of PRs a day. How do you solve the attribution problem for any one engineer's work given some set of impact metrics?
And keep in mind, most common impact metrics are trailing indicators, often over relative long time horizons.
Earlier quoted context omitted.
Because PDFs are a nightmare of a format and the only thing that’s is reasonably guaranteed about them is they will render to an image that people can read, the parsing of which will be much less token efficient than the equivalent text
I agree with you, but every non-engineer I know using these tools 100% will drag and drop a PDF into a chatbot. Anthropic and OpenAI as companies who are selling their products to all sorts of businesses should have a much better means of handling this nightmare of a format because it is so pervasive and so obviously what so many of their customers are going to drop into the product.
employees consumed 73.7 trillion tokens in roughly 30 days
, a figure tracked on an internal leaderboard called "Claudeonomics" — a reference to Anthropic's Claude, one of the third-party AI tools widely used inside the company [2]. The leaderboard, which ranked employees and teams by token consumption, inadvertently incentivized usage volume over productive output.
Meta plans to dismantle the leaderboard and replace it with a centralized monitoring platform called "AI Gateway," which will track usage and spending across teams in real time [2]."
This seems to be an interesting upcoming business, that is:
Helping companies centralize and track their AI usage by employee.
Anyway, great article!
I’d be curious to see the breakdown on spending by use case. I’ve heard it said that the majority of tokenmaxing comes from none technical uses like reading PDFs, creating PowerPoints, generating graphics/images… ect. But I’ve never heard any actual proof to that.
One thing I find fascinating as a software engineer who talks to non software engineers who use AI tools is how "reading PDFs" is not more of a solved problem. What I mean is that uploading a PDF into a chatbot tool seems to be an extraordinarily obvious use case that non technical (and technical) users would want to do. IMO claude, chatgpt/codex, etc should be able to optimize the PDF use case to be extremely token…
This and replies to this are surreal. It's like everyone simultaneously decided to forget that you don't need claude or whatever to read a PDF. The document is literally made for you to read...
Earlier quoted context omitted.
Because PDFs are a nightmare of a format and the only thing that’s is reasonably guaranteed about them is they will render to an image that people can read, the parsing of which will be much less token efficient than the equivalent text
I agree with you, but every non-engineer I know using these tools 100% will drag and drop a PDF into a chatbot. Anthropic and OpenAI as companies who are selling their products to all sorts of businesses should have a much better means of handling this nightmare of a format because it is so pervasive and so obviously what so many of their customers are going to drop into the product.
Token economics also are weird. If you design a fancy new frontend that for example uses a cheap model to parse a PDF into text that is fed into an expensive model, you will probably spend more money because you are on API payscale rather than the "max plan" payscale.
Earlier quoted context omitted.
> measure outcomes (impact) This is also not easy. In particular proactively preventing bugs is not rewarded
> In particular proactively preventing bugs is not rewarded The main way I think you can proactively prevent bugs in a meaningful way is by crafting and propagating better architecture. Better (or worse) architecture and adoption of it can be measured through a mix of quantitative and qualitative means so those metrics could be used to evaluate the impact of the engineer driving that architecture.
The engineer who haphazardly launched on Friday then promptly saved the team at 3am and worked the weekends gets the promotion, while the one who prevented a bug from happening "didn't get anything done" and gets the PIP.
I’d be curious to see the breakdown on spending by use case. I’ve heard it said that the majority of tokenmaxing comes from none technical uses like reading PDFs, creating PowerPoints, generating graphics/images… ect. But I’ve never heard any actual proof to that.
"Now he tells me!"
bonk