151–160 of 162 posts
It's stories like this that really dispell the genius/merit theory of successful business. The best you can say about Zuck is he didn't prevent Facebook from becoming huge.
Doesn't the legend go Zuck didn't even see the big picture until Sean Parker spelled it out for him?
Within a few weeks of telling people at our company that if they don’t use AI they will be replaced by someone who does, they just announced that their allocation with ChatGPT has reset and are now panicking as they blew through their million token allocation for this month in under six hours - you can’t make this shit up.
A million tokens is like $15 with SOTA models… that’s their allocation?
Earlier quoted context omitted.
I wonder if this should be codified as a rule of thumb, or an unofficial "law"? One perhaps we can reference easily among our peers.
You mean like Goodhart's Law? (Unless you're being sarcastic)
That says that when a measure becomes a target, it ceases to be a good measure.
My argument is that in the 21st century, a time of great data use without deep understanding, any measure at all will inherently become a goal to be gamed.
Earlier quoted context omitted.
I love how confidently you say this, with no evidence provided (and I doubt you have any.) Just a pristine comment section yap.
If there was a positive return on token spend they wouldn’t be capping it now would they?
In addition, as other commenters have pointed out, diminishing returns is a real phenomenon.
Earlier quoted context omitted.
> how "reading PDFs" is not more of a solved problem This and replies to this are surreal. It's like everyone simultaneously decided to forget that you don't need claude or whatever to read a PDF . The document is literally made for you to read...
> The document is literally made for you to read... It’s disingenuous to assume every PDF is actually crafted to communicate to its recipients, even more so to pretend LLM users are in a position to understand all the PDFs they receive There’s a lot of gray area where help understanding a document is fully reasonable
Earlier quoted context omitted.
Ok, uptime. How do you measure an individual’s contribution to uptime? If Claude goes down does everyone take a hit? If Claude stays up everyone gets rewarded? If so, your metric cannot distinguish between a bad engineer and a good one. If not, you have the same problem you started with: measuring contributions to “uptime”.
> If so, your metric cannot distinguish between a bad engineer and a good one. A metric that moves in the same direction and amount for everyone based on external event isn’t a problem. The delta in performance of the great engineer will outweigh that of the poor, since the metric movement that is due to external circumstances will be the same in each kind of engineer and thus not count.
Earlier quoted context omitted.
Because PDFs are a nightmare of a format and the only thing that’s is reasonably guaranteed about them is they will render to an image that people can read, the parsing of which will be much less token efficient than the equivalent text
For anyone needing to do this, the answer is to convert it to an image first. Far smaller, LLMs work well with them (even in some pretty insane use cases I've seen), and, along with human review, it can be a huge productivity gain that results in structured data.
Earlier quoted context omitted.
That does not follow. Many things have diminishing returns curves.
Yes, but a curve trends to zero. A cap is a step function to zero. I think they're saying there's no point in removing those diminishing return gains if there's still positive value to be had.
"The leaderboard, which ranked employees and teams by token consumption, inadvertently incentivized usage volume over productive output." Who could possibly have predicted that happening?
My company has an AI leaderboard and ONE ranking to be this, AND OTHER rankings like efficiency (loc merged per token). No one is so stupid that they think any single one of this is to be optimized (gamed) for. Tech journalists have low opinion of people with actual skills who actually contribute to society, and when their opinions get posted here, it's often selectively echoed by people looking for a reason to feel…
Earlier quoted context omitted.
You have to realize that if you set a measure, you're actually setting a goal for your employees. There is no such thing as a meaningless metric; why else would you measure it? No amount of "this isn't used for anything" will change that. It's inherent in human nature in the 21st century to believe any and all metrics will be used against them, and therefore must be gamed. It's why you also have to set UNBELIEVABLY c…
I worked at a place that argued that nobody would game the metrics because it would be wrong and they never stated what the metrics were… while I was gaming the metrics and they were praising me for being one of the best on the team. It was an unreal experience.