I’ve seen the same thing using AI for coding. It helps a lot at first, but after a while it starts doing weird stuff or undoing things that were already fixed. Now I treat it like a junior dev. I try to keep tasks small, reset often, and check everything. Still useful, just needs babysitting.
Is there a half-life for the success rates of AI agents?
121–130 of 143 posts
Re: Is there a half-life for the success rates of AI agents?
#122This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…
They poison their own context. Maybe you can call it context rot, where as context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly. Even with good context the rot will start to become apparent around 100k tokens (with Gemini 2.5). They really need to figure out a way to delete or "forget" prior context, so the user or even the model can go back and prune…
I often think that context should be a tree, where you can say - lets prune this whole branch, it didn't lead to anything good.
Re: Is there a half-life for the success rates of AI agents?
#123Earlier quoted context omitted.
They poison their own context. Maybe you can call it context rot, where as context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly. Even with good context the rot will start to become apparent around 100k tokens (with Gemini 2.5). They really need to figure out a way to delete or "forget" prior context, so the user or even the model can go back and prune…
I think you just coined "context rot", what an excellent term! Quoted you on my blog https://simonwillison.net/2025/Jun/18/context-rot/
maybe someday soon, LLMs will learn to smoothly forget their own irrelevant context
imagine instead of predicting just the next token, the LLM predicts a mask over the previous tokens, that is then thresholded and only “relevant” tokens are kept in the next inference
one key distinction between humans and LLMs is that humans are excellent at forgetting irrelevant data. we forget tens of times a second and only keep what's necessary
Re: Is there a half-life for the success rates of AI agents?
#124The amusing things LLMs do when they have been at a problem for some time and cannot fix it: - Removing problematic tests altogether - Making up libs - Providing a stub and asking you to fill in the code
| Making up libs
This is an attack vector. Probe the models for commonly hallucinated libraries (on npm or github or wherever) and then go and create those libraries with malicious code.
Re: Is there a half-life for the success rates of AI agents?
#125Re: Is there a half-life for the success rates of AI agents?
#126This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…
I got an idea. Context compression. Once the context reaches a certain size threshold have the LLM summarize it into bulletpoints then start a new session with that summary as the context.
Humans as well don’t remember the entire context either. For your case the summary already says tried library A and B and it didn’t work, it’s unlikely the LLM will repeat library A given that the summary explicitly said it was attempted.
I think what happens is that if the context gets to large the LLM sort of starts rambling or imitating rambling styles it finds online. The training does not focus on not rambling and regurgitation so the LLM is not watching too hard for that once the context gets past a certain length. People ramble too and we repeat shit a lot.
Re: Is there a half-life for the success rates of AI agents?
#127This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…
I had a particularly hard parsing problem so I setup a bunch of tests and let the LLM churn for a while and did something else. When I came back all the tests were passing! But as I ran it live a lot of cases were still failing. Turns out the LLM hardcoded the test values as “if (‘test value’) return ‘correct value’;”!
Seems like property based tests would be good for llms, it’s a shame that half the time coming up with a good property test can be as hard as writing the code.
Re: Is there a half-life for the success rates of AI agents?
#128My prediction is that the general limitation of multi-step agents is the quality of the reward function. If you think of LLMs as throwing shit at the wall and see if it sticks, but unlike traditional bruteforce (which is ”random” in the output space) we have a heuristic guided search with much higher expected results. But, even a well guided heuristic tapers off to noise after many steps, without checkpoints or corrections in global space. This is why AI crush most games, but gets a digital stroke and goes in circles during complicated problems.
Anyway, what this means I think is you will find AI agents continuing to colonize spaces with meaningful local and global reward functions. But most importantly, it likely means that complex problem spaces will see marginal improvements (where are all these new math theorems we were promised many months ago?).
It’s also very tempting to say ”ah but we can just make or even generate reward functions for those problems and train the AI”. I suspect this won’t happen, because if there was simple functions, we’d have discovered them already. Software engineering is one such mystery, and the reason I love it. Every year, we come up with new ideas and patterns. Many think they will solve all our problems, or at least consistently guide as in the right direction. But yet, here we are, debating language features, design patterns, tooling, UX etc etc. The vast majority of easy truths are already found. The rest are either complex or hard to find. Even when we think we found one, it often takes man-decades to conclude that it wasn’t even a good idea. And they’re certainly not inferrable from existing training data.
Re: Is there a half-life for the success rates of AI agents?
#129Interesting. So if you project outwards a while, you hit around 10000 hours about 6 years from now. Is that a reasonable timeline for ASI? It's got more of a rationale behind it than other methods perhaps?
10,000 hours of what are we comparing ? . Compilers already do better than me million hours for every program I write because I am not crafting assembly code. Computers save billions of hours compared to doing it by hand with an abacus or pen and paper. Productivity of humans is always dependent on tools they have access to, even with agents become that much productive so will humans who use tools —— Projecting doubl…
Great writeup. There are other scaling axes of course, around data (even synthetic data) and improving AI generation at the 'software' layer (smarter design / training efficiencies / inference speed ups) — progress in those might make the the currently-unthinkable orders of magnitude $500b and beyond not as necessary?
Re: Is there a half-life for the success rates of AI agents?
#130As long as LLMs have no true memory, this is expected. Think about the movie Memento. That is the experience for an LLM. What could any human do with a context window of 10 minutes and no other memory? You could write yourself notes… but you might not see them because soon you won’t know they are there. So maybe tattoo them on your body… You could likely do a lot of things. Just follow a recipe and cook. Drive to wor…
> I think solving memory is solving agi.
But we already have AGI