Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is there a half-life for the success rates of AI agents?

tobyord.com

Post reply on HN
Print view View original

101–110 of 143 posts

Re: Is there a half-life for the success rates of AI agents?

#101

Post by Aeolun »

drewnick wrote: post #87
libraryofbabel wrote: post #52

Earlier quoted context omitted.

Also in Claude Code you can just press a bunch of times and you can backtrack to an earlier point in the history before the context was poisoned, and re-start from there. Claude has some amazing features like this that aren’t very well documented. Yesterday I just learned it writes sessions to disk and you can resume them where you left off with -continue or - resume if you accidentally close or something.

Thank you! This just saved me after closing laptop and losing a chat in VS Code. Cool feature and always a place where Clause Code UX was behind chat - being able to see history. "/continue" saved me ~15 minutes of re-establishing the planning for a new feature. Also loving the shift + tab (twice) to enter plan mode. Just adding here in case it helps anyone else.

Claude code should really document this stuff in some kind of tutorial. There’s too much about code that I need to learn from random messages on the internet.

Re: Is there a half-life for the success rates of AI agents?

#102

Post by [deleted] »

jappgar wrote: post #95
kridsdale1 wrote: post #56

Earlier quoted context omitted.

Human health follows this principle too. N is the LifeSpan. The steps taken are cell division. Eventually enough problems accumulate that it fails systemically. Sexual reproduction is context-clearing and starting over from ROM.

This is precisely why there is not and never will be a fountain of youth. Sure, you could be cloned, but that wouldn't be you. The process of accumulating memories is also the process of aging with death being an inevitability. Software is sort of like this too, hence rewrites.

[deleted]

Re: Is there a half-life for the success rates of AI agents?

#103

Post by iLoveOncall »

Workaccount2 wrote: post #16
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

They poison their own context. Maybe you can call it context rot, where as context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly. Even with good context the rot will start to become apparent around 100k tokens (with Gemini 2.5). They really need to figure out a way to delete or "forget" prior context, so the user or even the model can go back and prune…

> They really need to figure out a way to delete or "forget" prior context

This is already pretty much figured out: https://www.promptingguide.ai/techniques/react

We use it at work and we never encounter this kind of issues.

Re: Is there a half-life for the success rates of AI agents?

#104

Post by jongjong »

I suspect it's because the average code online is flawed. The flaws are trained into LLMs and this manifests as an error rate which compounds.

I saw some results showing that LLMs struggle to complete tasks which would take longer than a day. I wonder if the average developer, individually, would be much better if they had to write the software on their own.

The average dev today is very specialized and their code is optimized for job security, not for correctness and not for producing succinct code which maps directly to functionality.

Re: Is there a half-life for the success rates of AI agents?

#105

Post by Aeolun »

peacebeard wrote: post #18
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

Very common to see in comments some people saying “it can’t do that” and others saying “here is how I make it work.” Maybe there is a knack to it, sure, but I’m inclined to say the difference between the problems people are trying to use it on may explain a lot of the difference as well. People are not usually being too specific about what they were trying to do. The same goes for a lot of programming discussion of c…

I ask it to build it to 3d voxel engine in Rust, and it just goes off and do it. Same for a vox file parser.

Sure, it takes some creative prompting, and a lot of turns to get it to settle on the proper coordinate system for the whole thing, but it goes ahead and does it.

This took me two days so far. Unfortunate, the scope of the thing is now so large that the quality rapidly starts to degrade.

Re: Is there a half-life for the success rates of AI agents?

#106

Post by Aeolun »

mtalantikite wrote: post #51
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

I had Claude Code deep inside a change it was trying to make, struggling with a test that kept failing, and then decided to delete the test to make the test suite pass. We've all been there! I generally treat all my sessions with it as a pairing session, and like in any pairing session, sometimes we have to stop going down whatever failing path we're on, step all the way back to the beginning, and start again.

Hah, I got “All this async stuff looks really hard, let’s just replace it with some synchronous calls”.

“Claude, this is a web server!”

“My apologies… etc.”

Re: Is there a half-life for the success rates of AI agents?

#108

Post by dr_dshiv »

Workaccount2 wrote: post #16
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

They poison their own context. Maybe you can call it context rot, where as context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly. Even with good context the rot will start to become apparent around 100k tokens (with Gemini 2.5). They really need to figure out a way to delete or "forget" prior context, so the user or even the model can go back and prune…

Context rot! Love it. One bad apple spoils the barrel.

I try to keep hygiene with prompts; if I get anything bad in the result, I try to edit my prompts to get it better rather than correcting in conversation.

Re: Is there a half-life for the success rates of AI agents?

#109

Post by manquer »

adverbly wrote: post #107

Interesting. So if you project outwards a while, you hit around 10000 hours about 6 years from now. Is that a reasonable timeline for ASI? It's got more of a rationale behind it than other methods perhaps?

10,000 hours of what are we comparing ? .

Compilers already do better than me million hours for every program I write because I am not crafting assembly code.

Computers save billions of hours compared to doing it by hand with an abacus or pen and paper.

Productivity of humans is always dependent on tools they have access to, even with agents become that much productive so will humans who use tools

——

Projecting doubling rate over so many generations is no different than saying earth is flat just because it feels flat in your local space. There is no reason to believe exponential doubling holds for 10 generations from now.

There is just one example in all of history that did that over 10 generations , Moore law . Doubling for so many generations at near constant rate is close to impossible , from the times of the ancient tale of grain and chessboard, people constants struggle with the power of the exponential.

——

I would say current LLM approaches are fairly close to end than beginning of the cycle, perhaps 1-2 generations left at best .

The outlay on funding is already at $50-$100 B/year per major player. No organization can spend $500B/year on anything. I don’t see any large scale collaboration in the private sector or in public like space station / fusion reactor for allocating resources to get to say a 3rd gen from now.

Comparing it with semiconductor tech the only other exponential example we have, the budget for foundries and research grew similarly and those are today are say $10-20B for leading edge foundry, but doesn’t keep growing at the same pace.

Constraints on capital availability and risk when deploying it is why we have so few fab players left and remaining players can afford to stagnate and not seriously invest .

Re: Is there a half-life for the success rates of AI agents?

#110

Post by simonw »

Workaccount2 wrote: post #16
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

They poison their own context. Maybe you can call it context rot, where as context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly. Even with good context the rot will start to become apparent around 100k tokens (with Gemini 2.5). They really need to figure out a way to delete or "forget" prior context, so the user or even the model can go back and prune…

I think you just coined "context rot", what an excellent term! Quoted you on my blog https://simonwillison.net/2025/Jun/18/context-rot/
Post reply on HN