Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

131–140 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#131

Post by dns_snek »

Herring wrote: post #123
habinero wrote: post #119

Earlier quoted context omitted.

That's not learning, that's building. It's like trying to learn how to draw via paint by numbers. Do you end up with something you could hang on the wall? Sure. Could you have fun doing it? Sure. Is there anything wrong with just doing that as a hobby? Of course not. Is it a substitute for actually learning how to look at objects and break them down into shapes and color and value? No. You gotta put in the work if yo…

I like the sentiment, I really do, but nobody (outside a phd program) pays you to learn. That's just not how society is set up. If FAANG companies could get away with hiring high school kids at min wage to prompt all day they would. We'll figure that out real quick as that exponential rises. If you don't like it, build a better society. While you still can.

Correction: Nobody wants to pay for you to learn, yet they implicitly do it and rely on it.

If companies decide that professional learning is unnecessary in the age of AI they'll be committing a horrible blunder. Their "fuck around" phase might sting, but missing an entire generation of skilled professionals is going to make our value skyrocket in the "find out" phase, a few years down the line.

Re: Measuring AI Ability to Complete Long Tasks

#132

Post by simonw »

Jcampuzano2 wrote: post #130
simonw wrote: post #11

I didn't really understand the "long task" thing until I actually experienced it. The problem is finding a task you can set an agent that justifies working for that long. I finally hit one when I tried porting that Python HTML5 parser to JavaScript by pointing Codex CLI at the 9,200 html5lib-tests test suite: https://simonwillison.net/2025/Dec/15/porting-justhtml/ It's pretty amazing to watch tools-in-a-loop crunch a…

How are you guys even doing long tasks with plain Codex or Claude code? I use Claude code and I get hit with a permissions prompt every 2 seconds for anything I try to do. Sure I can turn off all dangerous permissions but it'd probably honestly stop and claim it's finished well before it actually is in most cases from my experience. To be fair I haven't tried codex so maybe it's better at this but I'm my experience a…

You have to use --yolo or --dangerously-skip-permissions options.

Thankfully the cloud versions (Claude Code for web, Codex Cloud) run like that already, and are relatively safe in that if anything goes wrong it happens on someone else's computer.

Re: Measuring AI Ability to Complete Long Tasks

#133

Post by simonw »

throwaway613745 wrote: post #128
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

Just speaking from personal experience but the struggle is what creates the learning. I learned refactoring patterns from Fowler's book. But when I tried to actually use them I still struggled. I didn't fully understand how the patterns worked until I actually tried (and failed) to use them a few times. You don't really internalize things until you understand what doesn't work just as much as what does. You don't lea…

This conversation isn't about building a vector database from scratch, it's about learning to integrate with an existing vector database.

Re: Measuring AI Ability to Complete Long Tasks

#134

Post by ModernMech »

eleventen wrote: post #16
ModernMech wrote: post #8

Earlier quoted context omitted.

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

Respectfully, I think I’m in a better position to decide a) what value this has to me and b) what I choose to learn vs just letting Opus deal with. You don’t have enough information to say if I’ve saved time because you don’t know what I’m doing or what my goals are.

Respectfully, a) I didn't say anything about what value this has to you but moreover...

b) you also don't have enough information to say if it's saved you time because the costs you will bear are in the future. Systems require maintenance, that's a fact you can't get rid of with AI. And often times, maintaining systems require more work than building them in the first place. Maintaining systems tends to require a deep understanding of how they work and the tradeoffs that were decided when they were built.

But you didn't build the thing, you didn't even design it as you left that up to Claude. That makes the AI the only thing on the planet that understands the system, but we know actually the AI doesn't understand anything at all. So no one understands the system you built, including the AI you used. And you expect that this whole process will have saved you time, while you play games?

I just don't see it working out that way, sorry. The artifact the AI spit out will eventually demand you pay the cost in time to understand it, or you will incur future costs for not understanding it as it fails to act as you expect. You'll pay either way in the end.

Re: Measuring AI Ability to Complete Long Tasks

#135

Post by simonw »

dns_snek wrote: post #126
simonw wrote: post #113

Earlier quoted context omitted.

If it is I'd very much like to learn more about the science. I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that? If not, what's different about asking an LLM?

Well that's a gross oversimplification of the process. Hunting for a missing semicolon is a basic mechanical task that doesn't require much thought. Engaging with an intellectual problem, trying to solve it one way, failing, reasoning through the process and the requirements, trying to discover a better way of solving something, going down some wrong paths, backtracking, merging diverging ideas and ultimately finding…

My argument here is that you can still do hard work that helps you learn while leaning on an LLM to help along the way.

There's a reason kids do better when assigned a 1-1 tutor. LLMs, used effectively, can have a similar effect. Probably a weaker effect although maybe it can be stronger since there's no shame involved in asking an LLM a question.

Re: Measuring AI Ability to Complete Long Tasks

#136

Post by oidar »

GaggiX wrote: post #4
Dwedit wrote: post #3

Opus is already the name of an audio codec.

Opus: "an artistic work, especially one on a large scale." The names Haiku, Sonnet, and Opus have not been chosen randomly.

And so much more intuitive than the OpenAI names for their models. I still don't get their naming scheme.

Re: Measuring AI Ability to Complete Long Tasks

#137

Post by throwaway613745 »

simonw wrote: post #133
throwaway613745 wrote: post #128

Earlier quoted context omitted.

Just speaking from personal experience but the struggle is what creates the learning. I learned refactoring patterns from Fowler's book. But when I tried to actually use them I still struggled. I didn't fully understand how the patterns worked until I actually tried (and failed) to use them a few times. You don't really internalize things until you understand what doesn't work just as much as what does. You don't lea…

This conversation isn't about building a vector database from scratch, it's about learning to integrate with an existing vector database.

The topic is basically irrelevant. I could just edit my post to change the two instances of "vector database" to "vector database integration" and nothing else would change about my point.

I could change the post to be about learning word-working by watching a robot build a shelf and nothing would change.

Re: Measuring AI Ability to Complete Long Tasks

#138

Post by lelanthran »

noosphr wrote: post #56
fragmede wrote: post #54

Earlier quoted context omitted.

it's $200/month for the "unlimited" plan.

It's amazing how fast your account hits usage limits.

I think GP was being sarcastic: they did say that the plans were "unlimited".

I read

    It's "unlimited"
and

    It's unlimited
quite differently.

Re: Measuring AI Ability to Complete Long Tasks

#139

Post by ModernMech »

weitendorf wrote: post #41
ModernMech wrote: post #8

Earlier quoted context omitted.

The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it. At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would ha…

You do learn how to control claude code and architect/orient things around getting it to deliver what you want. That's a skill that is both new and possibly going to be part of how we work for a long time (but also overlaps with the work tech leads and managers do). My proto+sqlite+mesh project recently hit the point where it's too big for Claude to maintain a consistent "mental model" of how eg search and the db sch…

Yeah, this is close to my experience with it as well. The AI spits out some tutorial code and it works, and you think all your problems are solved. Then in working with the thing you start hitting problems you would have figured out if you had built the thing from scratch, so you have to start pulling it apart. Then you start realizing some troubling decisions the AI made and you have to patch them, but to do so you have to understand the architecture of the thing, requiring a deep dive into how it works.

At the end of the day, you've spent just as much time gaining the knowledge, but one way was inductive (building it from scratch) while the other is deductive (letting the AI build it and then tearing it apart). Is one better than the other? I don't know. But I don't think one saves more time than the other. The only way to save time is to allow the thing to work without any understanding of what it does.

Re: Measuring AI Ability to Complete Long Tasks

#140

Post by lelanthran »

simonw wrote: post #74
hatefulheart wrote: post #62

Earlier quoted context omitted.

Simon have you got to the point where you just don’t read the article? Others have pointed out your interpretation of long task is not the same as the article. Maybe this is the negative effects of excessive LLM usage that are spoken about.

They were right. I hadn't read enough of the article to understand what was meant by multi-hour tasks. I upvoted them for pointing that out.

>> Maybe this is the negative effects of excessive LLM usage that are spoken about.

> I upvoted them for pointing that out.

I'm also curious about what you think about the GPs question. TBH, responding after reading half an article was a common thing for most people pre-LLM anyway.

Post reply on HN