Measuring AI Ability to Complete Long Tasks • Hacker News Board

Re: Measuring AI Ability to Complete Long Tasks

#11

Post by simonw » Sun, Dec 21, 2025, 4:50 AM UTC

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/

It's pretty amazing to watch tools-in-a-loop crunch away for >4 hours to solve a generally difficult problem through sheer brute-force.

Re: Measuring AI Ability to Complete Long Tasks

#12

Post by OxfordOutlander » Sun, Dec 21, 2025, 4:53 AM UTC

ModernMech wrote: post #8

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

> inevitably Claude reaches a point where it can't help.

Perhaps not. If LLMs keep getting better, more competent models can help him stay on top of it lol.

Re: Measuring AI Ability to Complete Long Tasks

#13

Post by pants2 » Sun, Dec 21, 2025, 4:55 AM UTC

Dwedit wrote: post #3

Opus is already the name of an audio codec.

Gemini is already the name of a Greek god, a constellation, a space mission, a crypto exchange, an astrological sign, a car, and a comic villain! How will we ever figure out which one someone is talking about?

Re: Measuring AI Ability to Complete Long Tasks

#14

Post by tokioyoyo » Sun, Dec 21, 2025, 4:56 AM UTC

vachina wrote: post #10

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what. Great for Opus because you’re now a captive customer.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from scratch, if needed.

Re: Measuring AI Ability to Complete Long Tasks

#15

Post by evklein » Sun, Dec 21, 2025, 4:58 AM UTC

OxfordOutlander wrote: post #12

ModernMech wrote: post #8

Earlier quoted context omitted.

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

> inevitably Claude reaches a point where it can't help. Perhaps not. If LLMs keep getting better, more competent models can help him stay on top of it lol.

You're still captive to a product. Which means that when CloudCo. increases their monthly GenAI price from $50/mo. to $500/mo., you're losing your service or you're paying. By participating in the build process you're giving yourself a fighting chance.

Re: Measuring AI Ability to Complete Long Tasks

#16

Post by eleventen » Sun, Dec 21, 2025, 4:58 AM UTC

ModernMech wrote: post #8

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

Respectfully, I think I’m in a better position to decide a) what value this has to me and b) what I choose to learn vs just letting Opus deal with. You don’t have enough information to say if I’ve saved time because you don’t know what I’m doing or what my goals are.

Re: Measuring AI Ability to Complete Long Tasks

#17

Post by Avicebron » Sun, Dec 21, 2025, 4:59 AM UTC

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

> I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature

Opus/Anthropic is hands down the best in my experience. But using it feels like intellectual fast food (they all are), I hate the fact that I can build something like a neatly presentable one off spa tool (ty Simon) when I'm barely paying attention. it feels unsatisfying to use.

EDIT: because I'm rambling, I like "AI" as much as the next guy, probably more because I was there before it turned into LLMs"R"US, but I also like(d) the practice of sitting around listening to music solving problems with Scala. I don't know why we've decided to make work less fun..

Re: Measuring AI Ability to Complete Long Tasks

#18

Post by simonw » Sun, Dec 21, 2025, 4:59 AM UTC

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn.

You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works.

My hunch is that the 30 minutes of focused learning spent with a custom-built version that solves your exact problem is as effective (or even more effective) than four hours spent mostly struggling to get something up and running and going down various rabbit holes of unrelated problem-solving.

Especially if realistically you were never going to carve out those four hours anyway.

Re: Measuring AI Ability to Complete Long Tasks

#19

Post by pugio » Sun, Dec 21, 2025, 5:00 AM UTC

Opus looks like a big jump from the previous leader (GPT 5.1), but when you switch from "50%" to "80%", GPT 5.1 still leads by a good margin. I'm not sure if you can take much from this - perhaps "5.1 is more reliable at slightly shorter stuff, choose Opus if you're trying to push the frontier in task length".

Re: Measuring AI Ability to Complete Long Tasks

#20

Post by simonw » Sun, Dec 21, 2025, 5:01 AM UTC

tokioyoyo wrote: post #14

vachina wrote: post #10

Earlier quoted context omitted.

Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what. Great for Opus because you’re now a captive customer.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

I don't buy it.

I think that could work, but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature.

Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them.

I expect that we'll learn over time that LLM-managed big ball of mud codebases are less valuable than LLM-managed high quality well architected long-term maintained codebases.