Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

21–30 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#21

Post by aabhay »

simonw wrote: post #18
eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t take. That makes dealing with bugs and issues later on a lot easier, but it also expands my awareness of the domain, and checks my hubris on thinking I know something, and makes it possible to reason about the system when doing things later on.

Of course, this kind of interactive deep engagement with a topic is fast becoming obsolete. But the essence to me of “knowing” is about doing and experiencing things, updating my bayesian priors dialectically (to put it fancily)

Re: Measuring AI Ability to Complete Long Tasks

#22

Post by Aperocky »

tokioyoyo wrote: post #14
vachina wrote: post #10

Earlier quoted context omitted.

Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what. Great for Opus because you’re now a captive customer.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

Recreating everything from scratch gets harder and the previous requirements will eventually not be met after sufficient number of them have been accumulated. AI would have no solution to this unless it iterate on the same code base, but since I've not seen evidence of architectural maintainability from AI, a project that are fully given to AI is bound to fail.

AI is still incredibly useful used in tandem, but have it implement full feature from one sentence usually lead to doom.

Re: Measuring AI Ability to Complete Long Tasks

#23

Post by tokioyoyo »

simonw wrote: post #20
tokioyoyo wrote: post #14

Earlier quoted context omitted.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

I don't buy it. I think that could work , but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature. Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them. I expect that we'll learn over time that LLM-managed big…

Fair enough. In my imagination, I can see people writing AI-first framework/architectures and a general trend for people to “migrate to such frameworks”, just like the push towards the microservices architectures in 2010s. A part of these frameworks would be “re-constructibility” by changing contracts in parts where it matters, and somehow the framework would make it easy for the LLM to discover such “parts”.

Honestly, i’m making stuff up, as I don’t think it’s feasible right now because of the context sizes. But given how fast things develop, maybe in a couple of years things might change.

Re: Measuring AI Ability to Complete Long Tasks

#24

Post by Aperocky »

I think the problem here is LLM eventually pollute its context window with so much of the current task that the larger picture or architectural sanity is forgotten in favor of the current task at hand.

And rarely is a software one and done, with a few round like this, the software architecture would have become schizophrenic. Combating this tendency usually require a lot of the work of these "long task" to be thrown away and more closely limiting what the AI is trying to do as they happen. The success of one "long task" is not necessarily a good thing!

Re: Measuring AI Ability to Complete Long Tasks

#25

Post by karimQuant »

The big issue is the 50%, if you switch to 80% it's much less. Now if you are in the wrong side of 50% given the task was 4hours. How much additional time to 4hours you need. repeat trying to get the task done 50%*50%->25% , 50%^4 -> 6.25%. the cost of bad luck is very high.

Re: Measuring AI Ability to Complete Long Tasks

#27

Post by pastel8739 »

Avicebron wrote: post #17
eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

> I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature Opus/Anthropic is hands down the best in my experience. But using it feels like intellectual fast food (they all are), I hate the fact that I can build something like a neatly presentable one off spa tool (ty Simon) when I'm barely paying attention. it feels unsatisfying to use. EDIT:…

“We” didn’t decide to make work less fun, others decided for us.

Re: Measuring AI Ability to Complete Long Tasks

#28

Post by wmf »

nrhrjrjrjtntbt wrote: post #26

Why measure in minutes and not tokens? Seems you could cheat by slowing the ai down.

They measure the time it takes a human to complete the task. They don't care how long the AI takes (although in practice it's much faster than human). Measuring tokens isn't a good idea because newer models can complete tasks using fewer tokens.

Re: Measuring AI Ability to Complete Long Tasks

#29

Post by simonw »

aabhay wrote: post #21
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t…

I agree that the only reliable way to learn is to put knowledge into practice.

I don't think that's incompatible with getting help from LLMs. I find that LLMs let me try so much more stuff, and at such a faster rate, that my learning pace has accelerated in a material way.

Re: Measuring AI Ability to Complete Long Tasks

#30

Post by ehnto »

simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

I think you might be misunderstanding the article actually, this is about AI solving tasks as measured by how long it takes a human to solve the task. The AI could potentially solve it much quicker, but the use of "human time to solve" is an attempt to create a metric that reveals long horizon complexity (as I understand it anyway).

It's interesting because like the article notes, AI is really smashing benchmarks, but actual usefulness in automation of thought work is proving much more elusive. I think that collective experience of AI just not being that useful, or as useful as benchmarks suggest it should be, is captured in this metric.

Post reply on HN