Hot take:
There should be an anti leaderboard that highlight people under a threshold. Not trying to learn how to use ai while working at a company like Amazon is almost certainly a bad thing, and cause for looking into why.
Amazon employees are "tokenmaxxing" due to pressure to use AI tools
161–170 of 268 posts
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#162Earlier quoted context omitted.
Thanks for the inside insight. I'm surprised how few comments are written with the prior that Amazon managers aren't stupid or uninformed about how incentives work. My guess would be that someone created the leaderboard without a lot of consultation with managers, and that some employees feel a competitive urge to try to "win" the leaderboard by burning tokens.
Your comment is the equivalent of stating, that Jeff Bezos and Andy Jassy, do not really know their employees are carrying around urine bottles.
Management is told its a homebrew. Very commonly used by developers all over the industry.
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#163Earlier quoted context omitted.
How you burn 300 requests in a day? From my Copilot usage Opus consumes surprisingly few requests to do a lot of stuff. It isn’t paying by token but instead by prompt or something.
If you are using subagents for asynchronous work, you can burn through 300 requests in a workday easily.
Copilot didn't charge for subagents. You could do an insane amount of work with dozens of subagents with a single request and a deep enough prompt to kick it off.
I setup entire virtual teams (Dev, QA, product, reviewers etc with the initiating model just acting as the agent manager to keep it's context minimal) to one-shot some stuff and it kept churning and making progress.
Those days are just about over with the change to token pricing but for a time....
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#164Earlier quoted context omitted.
My questions for that approach are: Why treat AI as a special technology that needs enterprise-scale exploration to come up with a useful application? And why not take the alternative approach of identifying the subset of people who have indeed found solid uses and spread their best practices around? The top-down approach to encouraging (mandating?) AI usage strikes me as infantilizing to the workers, who are perfect…
Human nature? In the early nineties, it was common for experienced electrical engineers to keep on using schematic entry digital design and look down on RTL and synthesis tools, despite that fact the latter was already way more productive. At some point, management had to put their foot down and force everyone to switch to using synthesis. It's not unreasonable to assume that many people are set in their ways and unw…
It is completely unreasonable to assume that. Tech people are so hungry for productivity gains that they regularly will defy management forbidding them from using a tool, because the tool is so good they feel they have to have it.
If LLMs truly are as good as their proponents say, engineers will use them even if management outright forbade it. The fact that people aren't using them, and have to be forced, is extremely strong evidence that they are not in fact that useful.
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#165The fact that management signed off on measuring AI use through token usage shows how incompetent management really is, including in allegedly technical conmpanies like Amazon. Tokenmaxxing was an entirely expected and rational response. IOW You measure employees in stupid ways, you're going to get stupid behaviour as a consequence.
One argument I have heard in favour of this is that management knew this would be a side effect, but that it's more important to have people engage with AI as much as possible simply to explore what is actually possible. You are effectively knowingly wasting money in the expectation that you might learn something useful that will be more valuable in the long run.
All so that they can lose this accumulated knowledge during the next round of layoffs.
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#166https://en.wikipedia.org/wiki/Poe's_law I was just joking about it a few days ago (I swear I didn't know Amazon was doing this) https://news.ycombinator.com/item?id=48079533
> That’s my latest joke — that we’ll have to pretend like we used the tools so they can feel validated they’ve spent all this money on hyped up technology. So, yes, it’s em-dashes and “it’s not just this, it’s that …” so they can hopefully leave us alone
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#167Earlier quoted context omitted.
I guess the only difference between this and your example is the concrete efficiency gain from RTL and synthesis tools versus dubious applications of AI. I do agree with the second point about pushing people to explore new ways of doing things though.
> dubious applications of AI Leaving aside the ethical aspects of using AI (not because they're not valid, because they're off topic for this discussion), in my line of work, the capabilities and productivity improvement of AI are staggering. Most of it is not writing the new code, which is but a small part of chip design, but everything else. I can't give a concrete work example, but here is an experiment that I ran…
> use cases related to root causing complex simulation failures.
That's a pretty interesting use case. I assume this is for RTL simulation given the thread, but how do you connect the output of the simulator to the AI?
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#168Earlier quoted context omitted.
Requests are such a weird metric. We have a token limit via Copilot (unless I'm misunderstanding our setup), and most of my "features" burn 1 to 2% of my token limit per month on 4.7. But I don't admin our plan, and I'm unsure what we actually git. Vscode just gives me a percentage of tokens remaining metric. One of the weirder things about all this is how arbitrary and non objective the billing structure seems. One…
Copilot is currently based on requests (1 prompt = 1 request, with multipliers for different models). At the beginning of June the billing structure will change to just be normal API cost. Your features are going to start burning 10-20% of your token limit using 4.7
At my employer, everybody who has an opinion that matters is convinced that all of the overages by the high users will be more than made up for by the people who barely use it.
Maybe they’re right. But it’s really hard to see how.
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#169Earlier quoted context omitted.
I would guess they are trying to maximize training data
If I was being rewarded for using more tokens, I would feed LLM output back into the model. That's probably not very useful training data.
I personally know two people who are doing exactly that after a mandate rolled out at their work, the measurement is "tokens spent" and since they weren't finding many cases that required a lot of tokens they simply started to run agent loops feeding each other.
Absurdly wasteful but Goodhart's Law almost never fails.
Re: Amazon employees are "tokenmaxxing" due to pressure to use AI tools
#170Earlier quoted context omitted.
> I can point it at an STM32 cubemx starter repo and ask for a feature My experience is it will attempt read from the wrong memory block resulting in garbadge. But that's a while ago so maybe LLMs have gotten better.
The AI labs have all released at least 3 new models each since december, things move very quickly.
I've been hearing "the new model is so much better than the one from 6 months ago" every few months since 2023. It's never been true to date, so please understand why I am skeptical that it suddenly became true this time.