Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

181–190 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#181

Post by rishabhaiover »

theptip wrote: post #177
rishabhaiover wrote: post #111

Earlier quoted context omitted.

I've practiced a healthy skepticism of the recent boom but I can't reason why the long horizon time wouldn't stretch to 8 hours or a week worth's of effort from next year. After Opus-4.5, governments and organizations should really figure out a path out of this storm because we're in it now.

Doubling time has been 7 months for a while, so you should expect 8h not 1 week next year.

Predictions over historical data in a landscape with fragile priors doesn't seem like a strong metric to me (it's a useful approximation at best)

Re: Measuring AI Ability to Complete Long Tasks

#182

Post by ehnto »

dwohnitmok wrote: post #180
theptip wrote: post #177

Earlier quoted context omitted.

Doubling time has been 7 months for a while, so you should expect 8h not 1 week next year.

It's significantly accelerated to 4 months since the beginning of 2025, which puts 1 week within reach if things stay on trend. But yes 7 months is the more reliable long-term trend.

Can we attribute the acceleration to something specific, that might not actually continue growth? For example, agentic coding and reasoning models seem to have made a huge leap in abilities, but wouldn't translate to an ongoing exponential growth.

Re: Measuring AI Ability to Complete Long Tasks

#183

Post by gtowey »

aabhay wrote: post #21
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This feels like the exactly wrong way to think about it IMO. For me “knowledge” is not the explicit recitation of the correct solution, it’s all the implicit working knowledge I gain from trying different things, having initial assumptions fail, seeing what was off, dealing with deployment headaches, etc. As I work, I carefully pay attention to the outputs of all tools and try to mentally document what paths I didn’t…

You could say that knowledge is understanding all the paths that won't solve your problem.

Re: Measuring AI Ability to Complete Long Tasks

#185

Post by iwontberude »

Applejinx wrote: post #103
extr wrote: post #61

Earlier quoted context omitted.

I think people fool themselves with this kind of thing a lot. You debug some issue with your GH actions yaml file for 45 minutes and think you "learned something", but when are you going to run into that specific gotcha again? In reality the only lasting lesson is "sometimes these kinds of yaml files can be finnicky". Which you probably already knew at the outset. There's no personal development in continually bashin…

Github Actions, web development, stuff like that, are terrible examples of where not to use AI. You can't really go to giant piles of technical debt and look to those for places to be human. It's soul-destroying. My concern would be that vibe coding will make those places of soul-less technical debt even deeper and deadlier. There will be nobody there, for generations of cruft. Where once the technical debt was made…

I have a friend that took over a project that another dev started that had literally hundreds of markdown documents in repo with things as insane as software for managing souls in a quantum immortality scheme.

Re: Measuring AI Ability to Complete Long Tasks

#186

Post by csomar »

simonw wrote: post #113
csomar wrote: post #105

Earlier quoted context omitted.

The struggle is how you learn. I think that’s pretty much established scientifically by now?

If it is I'd very much like to learn more about the science. I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that? If not, what's different about asking an LLM?

I am far from being an expert on this topic. I took this course (many years ago?) https://www.coursera.org/learn/learning-how-to-learn and held this idea since.

> ind it hard to believe that wasting hours hunting for a missing semicolon

That's not what I meant by "struggle" and you do not use an LLM for that anyway.

Re: Measuring AI Ability to Complete Long Tasks

#187

Post by NiloCK »

NiloCK wrote: post #85

I appreciate horizon expansion as a fundamental metric, but duration seems like too crude a measure. We used to like it when computers were fast. An infinitely unscrupulous model provider could double this five hour result by cutting your output tokens/second in half! This isn't only a question of gaming the metric: the very strong current small-fast models (4.5 Haiku, Gemini 3 Flash) have no hope of being measured f…

Wow that was a garbage comment!

My introduction to this type of model measuring came from an interview where the repeatedly hammered-home point was that Sonnet 4.0 nailed a gigantic refactor (conversion of a large legacy asp.net or similar into react server-side components or similar) in a loop whose runtime was some large number of hours. I mistakenly attributed the same framing here.

Re: Measuring AI Ability to Complete Long Tasks

#188

Post by exe34 »

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I like having the flexibility. If it's something I want to learn, I'll ask it to write some explanation into an md that I can read, and I can also look at the code diff in more detail. but if it's tedious things like interacting with the android sdk, I'll just let it do whatever it needs to do to get the feature working.

Re: Measuring AI Ability to Complete Long Tasks

#189

Post by simonw »

csomar wrote: post #186
simonw wrote: post #113

Earlier quoted context omitted.

If it is I'd very much like to learn more about the science. I find it hard to believe that wasting hours hunting for a missing semicolon (at the very real risk of quitting entirely) is essential for learning. Does that mean every student who asks a TA or fellow-student to help them find that semicolon is hurting themselves when they do that? If not, what's different about asking an LLM?

I am far from being an expert on this topic. I took this course (many years ago?) https://www.coursera.org/learn/learning-how-to-learn and held this idea since. > ind it hard to believe that wasting hours hunting for a missing semicolon That's not what I meant by "struggle" and you do not use an LLM for that anyway.

What did you mean by struggle?

I use LLMs to help me spot mistakes like that all the time, and I encourage people learning to code to do the same.

Re: Measuring AI Ability to Complete Long Tasks

#190

Post by leecommamichael »

big-chungus4 wrote: post #184

"Train adversarially robust image model" is not a long task imo

I read their citations (which are actually the same authors of this paper) and they also define using Python's built-in web server to "build a web server" as a long task.
Post reply on HN