Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is there a half-life for the success rates of AI agents?

tobyord.com

Post reply on HN
Print view View original

61–70 of 143 posts

Re: Is there a half-life for the success rates of AI agents?

#61

Post by ge96 »

kridsdale1 wrote: post #56
PaulHoule wrote: post #13

This was always my mental model. If you have a process with N steps where your probability of getting a step right is p, your chance of success is pᶰ, or 0 as N → ∞. It affects people too. Something I learned halfway through a theoretical physics PhD in the 1990s was that a 50-page paper with a complex calculation almost certainly had a serious mistake in it that you'd find if you went over it line-by-line. I thought…

Human health follows this principle too. N is the LifeSpan. The steps taken are cell division. Eventually enough problems accumulate that it fails systemically. Sexual reproduction is context-clearing and starting over from ROM.

damn you telemorase

Re: Is there a half-life for the success rates of AI agents?

#62

Post by kridsdale1 »

__MatrixMan__ wrote: post #26

I don't think this has anything to do with AI. There's a half life for success rates.

It’s applicable here since people are experimenting with Agent Pipelines and so the existing literature on that kinda of systems engineering and robustness is useful to people who may not have needed to learn about it before.

Re: Is there a half-life for the success rates of AI agents?

#63

Post by reactordev »

mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

This happens in real life too when a dev builds something using too much copy pasta and encounters a build error. Stackoverflow was founded on this.

Re: Is there a half-life for the success rates of AI agents?

#64

Post by furyofantares »

deadbabe wrote: post #5

This is another reason why there’s no point in carefully constructing prompts and contexts trying to coax the right solution out of an LLM. The end result becomes more brittle with time. If you can’t zero shot your way to success the LLM simply doesn’t have enough training for your problem and you need a human touch or slightly different trigger words. There have been times where I’ve gotten a solution with such a mi…

This is half right, I think, and half very wrong. I always tell people if they're arguing with the LLM they're doing it wrong and for sure part of that is there's things they can't do and arguing won't change that. But the other part is it's hard to overstate their sensitivity to their context; when you're arguing about something it can do, you should start over with a better prompt (and, critically, no polluted context from its original attempt.)

Re: Is there a half-life for the success rates of AI agents?

#65

Post by EnPissant »

mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

Where you using Claude Code or something else? I've had very good luck with Claude Code not doing what you described.

Re: Is there a half-life for the success rates of AI agents?

#66

Post by kazinator »

kossae wrote: post #29
Workaccount2 wrote: post #16

Earlier quoted context omitted.

They poison their own context. Maybe you can call it context rot, where as context grows and especially if it grows with lots of distractions and dead ends, the output quality falls off rapidly. Even with good context the rot will start to become apparent around 100k tokens (with Gemini 2.5). They really need to figure out a way to delete or "forget" prior context, so the user or even the model can go back and prune…

This is my experience as well, and for now comes down to a workflow optimization. As I feel the LLM getting off track, I start a brand new session with useful previous context pasted in from my previous session. This seems to help steer it back to a decent solution, but agreed it would be nice if this was more automated based off of user/automated feedback (broken unit test, "this doesn't work", etc.)

"Human Attention to the Right Subset of the Prior Context is All You Need"

Re: Is there a half-life for the success rates of AI agents?

#67

Post by mikeocool »

vadansky wrote: post #23
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

I had a particularly hard parsing problem so I setup a bunch of tests and let the LLM churn for a while and did something else. When I came back all the tests were passing! But as I ran it live a lot of cases were still failing. Turns out the LLM hardcoded the test values as “if (‘test value’) return ‘correct value’;”!

Yeah — I had something like this happen as well — the llm wrote a half decent implementation and some good tests, but then ran into issues getting the tests to pass.

It then deleted the entire implementation and made the function raise a “not implemented” exception, updated the tests to expect that, and told me this was a solid base for the next developer to start working on.

Re: Is there a half-life for the success rates of AI agents?

#68

Post by aprilthird2021 »

skerit wrote: post #9
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

> I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a good result. Is this with something like Aider or CLine? I've been using Claude-Code (with a Max plan, so I don't have to worry about it wasting tokens), and I've had it successfully handle tasks that take over an hour. But getting there isn't super easy, that's true. The instructions/CLAUDE.md file need…

You have to give us more about your example of a task that takes over an hour with very detailed instruction. That's very intriguing

Re: Is there a half-life for the success rates of AI agents?

#69

Post by Wowfunhappy »

mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

> I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a good result.

I absolutely have, for what it's worth. Particularly when the LLM has some sort of test to validate against, such as a test suite or simply fixing compilation errors until a project builds successfully. It will just keep chugging away until it gets it, often with good overall results in the end.

I'll add that until the AI succeeds, its errors can be excessively dumb, to the point where it can be frustrating to watch.

Re: Is there a half-life for the success rates of AI agents?

#70

Post by [deleted] »

vadansky wrote: post #23
mikeocool wrote: post #2

This very much aligns with my experience — I had a case yesterday where opus was trying to do something with a library, and it encountered a build error. Rather than fix the error, it decided to switch to another library. It then encountered another error and decided to switch back to the first library. I don’t think I’ve encountered a case where I’ve just let the LLM churn for more than a few minutes and gotten a go…

I had a particularly hard parsing problem so I setup a bunch of tests and let the LLM churn for a while and did something else. When I came back all the tests were passing! But as I ran it live a lot of cases were still failing. Turns out the LLM hardcoded the test values as “if (‘test value’) return ‘correct value’;”!

[deleted]
Post reply on HN