Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

41–50 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#41

Post by weitendorf »

ModernMech wrote: post #8
eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

You do learn how to control claude code and architect/orient things around getting it to deliver what you want. That's a skill that is both new and possibly going to be part of how we work for a long time (but also overlaps with the work tech leads and managers do).

My proto+sqlite+mesh project recently hit the point where it's too big for Claude to maintain a consistent "mental model" of how eg search and the db schemas are supposed to be structured, kept taking hacky workarounds by going directly to a db at the storage layer instead of the API layer, etc. so I hit an insane amount of churn trying to get it to implement some of the features needed to get it production ready.

Here's the whackamole/insanity documented in git commit history: https://github.com/accretional/collector/compare/main...feat...

But now I know some new tricks and intuition for avoiding this situation going forward. Because I do understand the mental model behind what this is supposed to look like at its core, and I need to maintain some kind of human-friendly guard rails, I'm adding integration tests in a different repo and a README/project "constitution" that claude can't change but is accountable for maintaining, and configuring it to keep them in context while working on my project.

Kind of a microcosm of startups' reluctance to institute employee handbook/kpis/PRDs followed by resignation that they might truly be useful coordination tools.

Re: Measuring AI Ability to Complete Long Tasks

#42

Post by simonw »

weitendorf wrote: post #31
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

Generally I agree with your takes and find them very reasonable but in this case I think your deep experience might be coloring your views a bit. LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!). I think problem…

I think it's very easy to harm your learning by leaning into LLMs.

What I don't believe is that it HAS to be like this. Maybe it's my natural optimism showing through here, but I'm confident it's possible to accelerate rather than slow down your learning progress with LLMs, if you're thoughtful about how you apply them.

An open question for me is how feasible it is to teach people how to teach themselves effectively using this new technology.

I have a core belief that everything is learnable, if people are motivated to learn. I have no idea how to help instill that motivation in people who don't yet have it though!

Re: Measuring AI Ability to Complete Long Tasks

#43

Post by BoiledCabbage »

tacitusarc wrote: post #39
simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

My problem with the OpenAI models (GPT5.2 in particular) recently is an extreme aversion to doing more than the smallest step in a task before asking for using input. Even if I explicitly instruct it to continue without input until the task is complete, it ignores the instruction. I cannot imagine GPT5.2 working on a task for more than 2 minutes, let alone 4 hours. I’m curious if you’ve run into this and figured out…

What agent framework are you using? It can differ from one to the next on the same model.

Re: Measuring AI Ability to Complete Long Tasks

#44

Post by emp17344 »

bentobean wrote: post #35

> We show that this metric has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. If true, how much of this is a result of: 1. Genuine technical advancement or: 2. Shoveling trillions of dollars into compute resources in order to service incoming LLM requests in a way that is completely unrealistic over the long term? In other words… are we talking about genuine…

I wonder how much of this stuff is attributable to true model advancement, or if it’s an improvement in the genetic harness? It’s impossible to separate strict model improvement from improvement in the associated tools.

Re: Measuring AI Ability to Complete Long Tasks

#45

Post by Avicebron »

fluidcruft wrote: post #38
Avicebron wrote: post #17

Earlier quoted context omitted.

> I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature Opus/Anthropic is hands down the best in my experience. But using it feels like intellectual fast food (they all are), I hate the fact that I can build something like a neatly presentable one off spa tool (ty Simon) when I'm barely paying attention. it feels unsatisfying to use. EDIT:…

I sort of disagree. It's somewhat like having hypercard again. You can build fun UI things and make machines do what you want them to do. You can care about the parts you want to care about and not sweat about the parts you don't want to learn in detail (yet). And Claude and codex make great guides/Sherpas. There are just too many parts involved to do anything. For example today I built a simple data collection app t…

>only wasted 5min of Claude's time rather than hours of mine

I mean will you (we) retain all that it did after a few months go by? You may say we don't need to, but that sounds a little shallow given we're both on HN. Do you remember Gatsby's criticism of "Summer People"?

Re: Measuring AI Ability to Complete Long Tasks

#46

Post by johnfn »

aabhay wrote: post #21
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t…

But how much of that time is truly spent on learning relevant knowledge, and how much of it is just (now) useless errata? Take vector search for an example. Pre-GPT, I would spend like an hour chasing down a typo, like specifying 1023 instead of 1024 or something. This sort of problem is now trivially solved in minutes by a LLM that fully understands the API surface area. So what exactly do I lose by not spending that hour chasing it down? It has nothing to do with learning vector search better, and an LLM can do it better and faster than I can.

Re: Measuring AI Ability to Complete Long Tasks

#47

Post by mediaman »

bentobean wrote: post #35

> We show that this metric has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. If true, how much of this is a result of: 1. Genuine technical advancement or: 2. Shoveling trillions of dollars into compute resources in order to service incoming LLM requests in a way that is completely unrealistic over the long term? In other words… are we talking about genuine…

Much of this is due to vastly better posttraining RL, not models that are much bigger. The idea that most of these gains comes from training really big models, or throwing immensely larger amounts of compute at it, is not really true.

Re: Measuring AI Ability to Complete Long Tasks

#48

Post by cornel_io »

simonw wrote: post #20
tokioyoyo wrote: post #14

Earlier quoted context omitted.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

I don't buy it. I think that could work , but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature. Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them. I expect that we'll learn over time that LLM-managed big…

And at the end of the day it's not really a tradeoff we'll need to make, anyways: my experience with e.g. Claude Code is that every model iteration gets much better at avoiding balls of mud, even without tons of manual guidance and pleading.

I get that even now it's very easy to let stuff get out of hand if you aren't paying close attention yourself to the actual code, so people assume that it's some fundamental limitation of all LLMs. But it's not, much like 6 fingered hands was just a temporary state, not anything deep or necessary that was enforced by the diffusion architecture.

Re: Measuring AI Ability to Complete Long Tasks

#49

Post by twotwotwo »

I'm conflicted about opining on models: no individual has actually done a large sample of real-world tasks with a lot of models to be able to speak with authority, but I kinda think we should each share our dubiously-informed opinions anyway because benchmarks aren't necessarily representative of real-world use and many can clearly be gamed.

Anyhow, I noticed more of a difference trying Opus 4.5 compared to Sonnet 4.5 than I'd noticed from, for example, the last couple Sonnet bumps. Objectively, at 1.66x Sonnet's price instead of the old 5x, it's much more often practical to consider reaching for than past Opus models. Anthropic's basic monthly thing also covers a fair amount of futzing with it in CC.

At the other extreme, another surprise of this family is that Haiku 4.5 with reasoning on is usable: better than Sonnet with thinking off according to some bencharks, and in any case subjectively decent for point edits, single-page thingies, and small tools.

Re: Measuring AI Ability to Complete Long Tasks

#50

Post by weitendorf »

tokioyoyo wrote: post #23
simonw wrote: post #20

Earlier quoted context omitted.

I don't buy it. I think that could work , but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature. Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them. I expect that we'll learn over time that LLM-managed big…

Fair enough. In my imagination, I can see people writing AI-first framework/architectures and a general trend for people to “migrate to such frameworks”, just like the push towards the microservices architectures in 2010s. A part of these frameworks would be “re-constructibility” by changing contracts in parts where it matters, and somehow the framework would make it easy for the LLM to discover such “parts”. Honestl…

No you're not making it up, this is exactly what some people are working on. Agent frameworks are starting to move towards "dynamic" service discovery/runtime introspection and composition-with-guardrails. Some keywords are "agent mesh", and the general marketing from AI companies about AI "inventors", and agent-driven interfaces like Google's a2ui (which is just a spec)

We recently started working on https://github.com/accretional/collector to serve as a dynamic proto ORM+CRUD server with search and discovery, and features for operating as a node in an "agent/service mesh". The idea is that you can create a uniform interface for data retrieval/search/APIs that lets agents dynamically register, invoke, or discover any data type or service, or write it themselves, then register it locally or share it.

It is feasible to do this stuff now actually, just a bit tricky because most LLMs aren't trained to operate this way without very explicit instructions for how to do so, and for collector specifically the API surface is probably too big. But I am pretty sure neither would take long to fix if enough people were adopting this kind of pattern.

Post reply on HN