Measuring AI Ability to Complete Long Tasks • Hacker News Board

Re: Measuring AI Ability to Complete Long Tasks

#111

Post by rishabhaiover » Sun, Dec 21, 2025, 2:24 PM UTC

ehnto wrote: post #30

simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

I think you might be misunderstanding the article actually, this is about AI solving tasks as measured by how long it takes a human to solve the task. The AI could potentially solve it much quicker, but the use of "human time to solve" is an attempt to create a metric that reveals long horizon complexity (as I understand it anyway). It's interesting because like the article notes, AI is really smashing benchmarks, bu…

I've practiced a healthy skepticism of the recent boom but I can't reason why the long horizon time wouldn't stretch to 8 hours or a week worth's of effort from next year. After Opus-4.5, governments and organizations should really figure out a path out of this storm because we're in it now.

Re: Measuring AI Ability to Complete Long Tasks

#112

Post by simonw » Sun, Dec 21, 2025, 2:28 PM UTC

girvo wrote: post #110

simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

> Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works I can only speak for myself, but the only way I've been able to learn things rapidly in this industry is by writing things myself: even rote re-typing of books or SO answers was enough to trigger this fo…

That's why I said "tweaking it yourself" - that's the point where you go beyond "just querying models and reading output".

Re: Measuring AI Ability to Complete Long Tasks

#113

Post by simonw » Sun, Dec 21, 2025, 2:30 PM UTC

csomar wrote: post #105

simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

The struggle is how you learn. I think that’s pretty much established scientifically by now?

If it is I'd very much like to learn more about the science.

I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that?

If not, what's different about asking an LLM?

Re: Measuring AI Ability to Complete Long Tasks

#114

Post by rich_sasha » Sun, Dec 21, 2025, 2:33 PM UTC

How does "cost" per frontier task change with time?

Extrapolating any exponential growth is always dangerous, but over say 3 years at this pace, we'd go from 2 hours to 70,or about 8 days' work.

Quite scary. But what does cost do over the same timeline? Does it increase with computational complexity? Is it worse - because, IIRC, transformers computational cost is quadratic in context length. Is it better - some kind of economies of scale?

I glanced thought the article but couldn't find any info on this.

Re: Measuring AI Ability to Complete Long Tasks

#115

Post by simonw » Sun, Dec 21, 2025, 2:35 PM UTC

risyachka wrote: post #104

simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

Reading without actually doing does not really result in learning, only very marginal one. Try reading tutorials on a new programming language for 30 minutes and then open new text file and write basic loop with print. It won’t even compile- which shows you haven’t really learned anything. Just read an interesting story. Sure you pita few bits here and there but you still don’t know how to do even the moat basic thin…

Working with an LLM feels very different to me from reading a static tutorial.

It's more like having the tutorial author there with you and actively engaging with them to collaborate on building the exact tutorial for the exact project you're looking to build.

I'll take a bidirectional conversation with a subject matter expert (and when you're just staring to learn Rust LLMs can absolutely take the role of "expert", in comparison to you at least) over struggling on my own against static documentation and the Rust compiler.

And I can take over the wheel at any moment! It's entirely on me to decide how much I get to do vs how much the LLM does for me.

That's why learning in this way is a skill in its own right, and one that I'd like to see studied and formalized and taught to other people.

Re: Measuring AI Ability to Complete Long Tasks

#116

Post by simonw » Sun, Dec 21, 2025, 2:37 PM UTC

Applejinx wrote: post #101

simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This really makes for a good natural experiment: carry on :) I have a hard time imagining how much you'd have to literally bribe me to get me to try doing it the way you describe. I'm too interested in implementation details of things and looking for innovations—in fact I make my living doing that, like some cyberpunk gremlin just delighting in messing with stuff in unexpected ways. I don't understand why you're not,…

I delight in messing with stuff in unexpected ways, and see that as a great way to learn. I just don't want to have to type all of that mischief out by hand.

I ported a complex new Python program I wrote to Go last night on my phone out of nothing more than wild curiosity. You can bet I learned a bunch about Go in the process, from bed, in about 20 minutes.

Re: Measuring AI Ability to Complete Long Tasks

#117

Post by yismail » Sun, Dec 21, 2025, 3:01 PM UTC

cubefox wrote: post #99

yismail wrote: post #7

Would be interesting to see Gemini 3.0 Pro benchmarked as well.

This article was published a long time ago, in March.

That's true, but it looks like it's been updated since then because the benchmarks include Claude Opus 4.5

Re: Measuring AI Ability to Complete Long Tasks

#118

Post by simonw » Sun, Dec 21, 2025, 3:07 PM UTC

lifis wrote: post #78

simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

Quickly looking at the source code, mostly treeBuilder and tokenizer, I do see several possible improvements: - Use Typescript instead of JavaScript - Use perfect hashes instead of ["a', "b", "c"].includes() idioms, string equalities, Seys, etc. - Use a single perfect hash to match all tags/attribute names and then use enums in the rest of the codebase - Use a single if (token.kind === Tag.START instead of repeating…

Thanks for the feedback - I pasted it into a Claude Code session on my phone, here's the resulting PR: https://github.com/simonw/justjshtml/pull/7

I didn't include the TypeScript bit though - it didn't use TypeScript because I don't like adding a build step to my JavaScript projects if I can possible avoid it. The agent would happily have used TypeScript if I had let it.

I don't like that openElements = open_elements pattern either - it did that because I asked it for a port of a Python library and it decided to support the naming conventions for both Python and JavaScript at once. I told it to remove all of those.

I had it run a micro benchmark too against the before and after - here's the code it used for that: https://github.com/simonw/justjshtml/blob/a9dbe2d7c79522a76f...

  BEFORE benchmark:
  Input: 87,707 bytes
  Average: 7.846 ms
  Ops/sec: 127.5

After applying your suggestions:

  AFTER: 
  Average: 7.769ms
  Ops/sec: 128.7  (1% improvement)

It pushed back against the tail recursion suggestion:

> The current implementation uses a switch statement in step(). JavaScript doesn’t have proper tail call optimization (only Safari implements it), so true tail recursion would cause stack overflow on large documents.

Re: Measuring AI Ability to Complete Long Tasks

#119

Post by habinero » Sun, Dec 21, 2025, 3:07 PM UTC

simonw wrote: post #40

gflarity wrote: post #33

Earlier quoted context omitted.

Consider, ever so briefly, that people don't all learn the same. You do you.

That's fair. Something I'm really interested right now is the balance in terms of the struggle required to learn something. I firmly believe that there are things where the only way to learn how to do them is to go through the struggle. Writing essays for example - I don't think you can shortcut learning to write well by having an LLM do that for you, even though actually learning to write is a painful and tiresome p…

That's not learning, that's building. It's like trying to learn how to draw via paint by numbers. Do you end up with something you could hang on the wall? Sure. Could you have fun doing it? Sure. Is there anything wrong with just doing that as a hobby? Of course not.

Is it a substitute for actually learning how to look at objects and break them down into shapes and color and value? No. You gotta put in the work if you want the result. Brains just work like that.

"Struggling with semicolons" isn't any different than drawing a hundred derpy looking faces that look terrible.

Ira Glass has a quote about this:

> “Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know its normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story. It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.”

Re: Measuring AI Ability to Complete Long Tasks

#120

Post by 0x000xca0xfe » Sun, Dec 21, 2025, 3:12 PM UTC

After spending many hours optimizing some routines I now think performance optimization is a great benchmark for identifiying how generally smart an AI is at helping with some specific piece of code.

Solutions are quite easy to verify with differential testing and produce a number for direct comparison.

Less code is usually better and you generally can't "cheat" by adding more cruft so it nullifies the additive bias. Good optimization requires significant understanding of the underlying structures. Everything has performance tradeoffs so it requires systemic thinking and not just stringing independent pieces together.

So far I've found that Gemini Pro 3 was the best at reasoning about tricky SIMD code but the results with most models were pretty underwhelming.