Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

51–60 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#51

Post by weitendorf »

tokioyoyo wrote: post #14
vachina wrote: post #10

Earlier quoted context omitted.

Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what. Great for Opus because you’re now a captive customer.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

It does matter because the code needs to still be legible and discoverable and semantic enough for other AI to find it and use it without it being so confusing or painful that they prefer to just write it themselves.

The reason software is so valuable is that it's capital/up-front investment in figuring something out that can continuously deliver value with low or no marginal cost. Rewriting/maintenance/difficulty/figuring out software is marginal cost.

Re: Measuring AI Ability to Complete Long Tasks

#52

Post by Arainach »

simonw wrote: post #42
weitendorf wrote: post #31

Earlier quoted context omitted.

Generally I agree with your takes and find them very reasonable but in this case I think your deep experience might be coloring your views a bit. LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!). I think problem…

I think it's very easy to harm your learning by leaning into LLMs. What I don't believe is that it HAS to be like this. Maybe it's my natural optimism showing through here, but I'm confident it's possible to accelerate rather than slow down your learning progress with LLMs, if you're thoughtful about how you apply them. An open question for me is how feasible it is to teach people how to teach themselves effectively…

> An open question for me is how feasible it is to teach people how to teach themselves effectively using this new technology.

It's not really an open question. We've had a huge amount of content on the internet including documentation, tutorials, example code, and actual online courses available for years and in the end most people don't learn effectively when presented with that information and left to themselves. LLMs are no different.

Re: Measuring AI Ability to Complete Long Tasks

#53

Post by noosphr »

simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

What's more amazing is how fast your account empties when they do that.

Re: Measuring AI Ability to Complete Long Tasks

#54

Post by fragmede »

noosphr wrote: post #53
simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

What's more amazing is how fast your account empties when they do that.

it's $200/month for the "unlimited" plan.

Re: Measuring AI Ability to Complete Long Tasks

#55

Post by tokioyoyo »

weitendorf wrote: post #50
tokioyoyo wrote: post #23

Earlier quoted context omitted.

Fair enough. In my imagination, I can see people writing AI-first framework/architectures and a general trend for people to “migrate to such frameworks”, just like the push towards the microservices architectures in 2010s. A part of these frameworks would be “re-constructibility” by changing contracts in parts where it matters, and somehow the framework would make it easy for the LLM to discover such “parts”. Honestl…

No you're not making it up, this is exactly what some people are working on. Agent frameworks are starting to move towards "dynamic" service discovery/runtime introspection and composition-with-guardrails. Some keywords are "agent mesh", and the general marketing from AI companies about AI "inventors", and agent-driven interfaces like Google's a2ui (which is just a spec) We recently started working on https://github.…

That’s actually really cool, and makes sense in my head! This is somewhat how I imagined it, except my guess would be someone would fine tune a general purpose LLMs (somehow, as it is much cheaper than starting from scratch, idk?) to behave this way rather than instructing it all the way in. And whoever develops the framework would package it with the access to this fine-tuned LLM.

But yeah, what you guys are doing looks sweet! I need to get out of my ass and see what people are doing in this sphere as it sounds fun.

Re: Measuring AI Ability to Complete Long Tasks

#57

Post by mmasu »

aabhay wrote: post #21
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t…

I remember a very nice quote from an Amazon exec - “there is no compression algorithm for experience”. The LLM might as well do wrong things, and you still won’t know what you don’t know. But then, iterating with LLMs is a different kind of experience; and in the future people will likely do that more than just grinding through the failure of just missing semicolons Simon is describing below. It’s a different paradigm really

Re: Measuring AI Ability to Complete Long Tasks

#58

Post by skeptic_ai »

simonw wrote: post #20
tokioyoyo wrote: post #14

Earlier quoted context omitted.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

I don't buy it. I think that could work , but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature. Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them. I expect that we'll learn over time that LLM-managed big…

It’s interesting how the monolith companies with a big ball of shit still stay in business.

But I’d say some projects (I expect to live less than 1 year) I’d just vibe code them so I won’t care much about the code. I just give very high level architectural ideas and that’s it.

Other projects which I expect lifespan to be more than 1-2 years I won’t let it become a ball of shit.

So it depends on the project.

Re: Measuring AI Ability to Complete Long Tasks

#59

Post by gizmodo59 »

pugio wrote: post #19

Opus looks like a big jump from the previous leader (GPT 5.1), but when you switch from "50%" to "80%", GPT 5.1 still leads by a good margin. I'm not sure if you can take much from this - perhaps "5.1 is more reliable at slightly shorter stuff, choose Opus if you're trying to push the frontier in task length".

Yeah. 50% of the time to throw away expensive tokens and limits is not ideal. But I bet by this time next year OSS models will be at that capability!

Re: Measuring AI Ability to Complete Long Tasks

#60

Post by weitendorf »

tokioyoyo wrote: post #55
weitendorf wrote: post #50

Earlier quoted context omitted.

No you're not making it up, this is exactly what some people are working on. Agent frameworks are starting to move towards "dynamic" service discovery/runtime introspection and composition-with-guardrails. Some keywords are "agent mesh", and the general marketing from AI companies about AI "inventors", and agent-driven interfaces like Google's a2ui (which is just a spec) We recently started working on https://github.…

That’s actually really cool, and makes sense in my head! This is somewhat how I imagined it, except my guess would be someone would fine tune a general purpose LLMs (somehow, as it is much cheaper than starting from scratch, idk?) to behave this way rather than instructing it all the way in. And whoever develops the framework would package it with the access to this fine-tuned LLM. But yeah, what you guys are doing l…

> fine tune a general purpose LLMs (somehow, as it is much cheaper than starting from scratch, idk?) to behave this way rather than instructing it all the way in

I'd love to do that too but there are basically three ways to teach LLMs how to use it afaik: with data created "in the wild" and a degree of curation or augmentation, or with full-on reinforcement learning/goal-oriented training, or some kind of hybrid based on eg conformance testing and validating LLM output at a less sophisticated level (eg if it tries to call an api that's not in the set that it just saw during discovery, the LLM is being dumb, train it out of doing that).

The thing is they are not really mutually exclusive, and LLM companies will do it anyway to make their models useful if enough people are using this or want to use it. This is what's happened already with eg MCP and skills and many programming languages. Anyway, if prompting works to get it to use it properly it validates that the model can be trained to follow that process too, the same way it knows how to work with React

Post reply on HN