Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

151–160 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#151

Post by hnthrowaway121 »

yoan9224 wrote: post #150

The key insight from this benchmark is using "human-equivalent hours" rather than actual AI execution time. It's measuring capability complexity, not speed. What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed. This is why I think the current "agent" paradigm nee…

You’ve only wasted the 4 hours if you didn’t spend them doing something else.

At 50/50 it’s an ok bet if the debugging time is much less than the total human time, even if the loops are long, you might rather 4 hours of deep work on an important human thing or on just relaxing vs babysitting the LLM. Assuming that about half the time that will pay off with a correctly done thing with very little effort, it’s kind of amazing.

Re: Measuring AI Ability to Complete Long Tasks

#152

Post by latentsea »

weitendorf wrote: post #41
ModernMech wrote: post #8

Earlier quoted context omitted.

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

You do learn how to control claude code and architect/orient things around getting it to deliver what you want. That's a skill that is both new and possibly going to be part of how we work for a long time (but also overlaps with the work tech leads and managers do). My proto+sqlite+mesh project recently hit the point where it's too big for Claude to maintain a consistent "mental model" of how eg search and the db sch…

I agree with this sentiment a lot. I find my experience matches this. It's not necessarily fast at first, but you learn lessons along the way that develop a new set of techniques and ways of approaching the problem that feel fundamental and important to have learnt.

My fun lesson this week was there's not a snowballs chance in hell GitHub Copilot can correctly update a Postman collection. I only realised there was a Postman MCP server after battling through that ordeal and eventually making all the tedious edits myself.

Re: Measuring AI Ability to Complete Long Tasks

#153

Post by afro88 »

yoan9224 wrote: post #150

The key insight from this benchmark is using "human-equivalent hours" rather than actual AI execution time. It's measuring capability complexity, not speed. What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed. This is why I think the current "agent" paradigm nee…

> The key insight from this benchmark is using "human-equivalent hours" rather than actual AI execution time. It's measuring capability complexity, not speed.

> What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed.

Your first two paragraphs are at odds with each other. If it fails, you've potentially wasted the time it took the agent to *perform* the "it takes humans 4h" long task. Which in most cases is single digit minutes.

That's why one of the solid use cases for agents is doing multiple throw away proof of concepts to explore a problem / new feature before deciding on a solution to actually implement. Usually you'd have time for one, or maybe none. If it fails you've lost a maybe 10 minutes, but likely learned something new about the potential solution.

Re: Measuring AI Ability to Complete Long Tasks

#154

Post by throwaway613745 »

simonw wrote: post #144
throwaway613745 wrote: post #137

Earlier quoted context omitted.

The topic is basically irrelevant. I could just edit my post to change the two instances of "vector database" to "vector database integration" and nothing else would change about my point. I could change the post to be about learning word-working by watching a robot build a shelf and nothing would change.

I genuinely do think you can learn 90% of what that is to learn about integrating with a vector database from having an LLM do the work for you and then carefully reviewing what it did. Turns out there's science that backs me up here: https://en.wikipedia.org/wiki/Worked-example_effect - showing people "worked examples" can be more effective than making them solve the problem themselves. That Wikipedia article is a l…

> Worked examples are step-by-step illustrations of the process required to complete a task or solve a problem.

That’s not what having a bot generate your integration is and reading it post-facto is. The bot isn’t guiding you through the process so you can go do it yourself. At best you would use this as a reference to go do another integration yourself - but at this point why even bother when you can just get the bot to do it again?

The only thing people learn using AI is how to do things with AI.

Re: Measuring AI Ability to Complete Long Tasks

#155

Post by jfreds »

simonw wrote: post #144
throwaway613745 wrote: post #137

Earlier quoted context omitted.

The topic is basically irrelevant. I could just edit my post to change the two instances of "vector database" to "vector database integration" and nothing else would change about my point. I could change the post to be about learning word-working by watching a robot build a shelf and nothing would change.

I genuinely do think you can learn 90% of what that is to learn about integrating with a vector database from having an LLM do the work for you and then carefully reviewing what it did. Turns out there's science that backs me up here: https://en.wikipedia.org/wiki/Worked-example_effect - showing people "worked examples" can be more effective than making them solve the problem themselves. That Wikipedia article is a l…

Having gone though exactly this exercise recently comparing a homegrown vector db against Qdrant, I’m wholeheartedly in agreement that getting a working solution FAST, and then spending a decent amount of time interrogating it (with help of LLM), is my favorite learning pattern

Re: Measuring AI Ability to Complete Long Tasks

#156

Post by stavros »

Jcampuzano2 wrote: post #130
simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

How are you guys even doing long tasks with plain Codex or Claude code? I use Claude code and I get hit with a permissions prompt every 2 seconds for anything I try to do. Sure I can turn off all dangerous permissions but it'd probably honestly stop and claim it's finished well before it actually is in most cases from my experience. To be fair I haven't tried codex so maybe it's better at this but I'm my experience a…

Codex (at least 5 and 5.1) is bad at asking for permission. Whenever it wants to run pre-commit or platformio, it tries to do that, that fails because of the sandbox, and then Codex decides something is wrong with the cache directory and keeps asking for permission to sudo chown ~/.cache, every time.

I have to specifically tell it to request permission for the command it wants to run, and then it works. Very annoying, and very annoying that it can't persist the permission, like Claude Code can, so it doesn't have to ask again every single time.

Re: Measuring AI Ability to Complete Long Tasks

#157

Post by DANmode »

mmasu wrote: post #70
visarga wrote: post #67

Earlier quoted context omitted.

Of course there is - if you write good tests, they compress your validation work, and stand in for your experience. Write tests with AI, but validate their quality and coverage yourself. I think the whole discussion about coding agent reliability is missing the elephant in the room - it is not vibe coding, but vibe testing. That is when you run the code a few times and say LGTM - the best recipe to shoot yourself in…

this is a very good point, however the risk of writing bad or non extensive tests is still there if you don’t know what good looks like! The grind will still need to be there, but it will be a different way of gaining experience

Starting to get it!

New skills, not no skills.

There will still be a wide spectrum of people that actually understand the stack - and don’t - and no matter how much easier or harder the tools get, those people aren’t going anywhere.

Re: Measuring AI Ability to Complete Long Tasks

#158

Post by DANmode »

weitendorf wrote: post #31
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

Generally I agree with your takes and find them very reasonable but in this case I think your deep experience might be coloring your views a bit. LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!). I think problem…

What would you have us do, though?

Stifle the tools, somehow?

You’ve had nontechnical devs since npm, or before!

No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.

Re: Measuring AI Ability to Complete Long Tasks

#159

Post by DANmode »

vachina wrote: post #10
eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what. Great for Opus because you’re now a captive customer.

If you don’t know that Opus isn’t an entity, but a model,

you might be a little too far removed from the situation to comment authoritatively?

Re: Measuring AI Ability to Complete Long Tasks

#160

Post by DANmode »

simonw wrote: post #145
fluidcruft wrote: post #107

Earlier quoted context omitted.

My experience with LLM and agents has led to the opinion that a LLM-friendly codebase is actually a very human friendly code base.

Same here. So far everything I have found to help LLMs is just good practice generally: automated tests, documentation, clear issue descriptions, a neat commit history, well featured code etc.

Documentation, aka any kind of architectural plan,

vs

“we’ll figure it out when we get there” human slop.

Post reply on HN