Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

191–196 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#191

Post by beasthacker »

simonw wrote: post #176
beasthacker wrote: post #173

Earlier quoted context omitted.

I buy the productivity argument, but I’m not convinced “30 minutes reading/tweaking agent output” is equivalent for learning to building it yourself. If your goal is the feature, then yes: letting the agent do the heavy lifting and reviewing the diff afterward is a huge win. But if your goal is understanding / skill-building, the hard part usually isn’t seeing a working solution. It’s doing the messy work of (a) maki…

If I had four hours to dedicate to this particular learning project I would still use LLMs to help me along the way, with the expectation that I'd learn more from those four hours than I would if I'd spent the same amount of time deliberately not using LLMs to help me. We've been given a tool that lets us ask questions in human language and get back answers that are correct 90% of the time! And that remaining 10% mea…

Hypothetical for you:

Learn more if you tried to figure it out yourself for 3 hours then used the LLM for the last hour to unblock/check your work? Or learn more by utilizing LLM for help the whole four hours?

My own experience is what I learn from an LLM sticks better if I take the former approach.

Re: Measuring AI Ability to Complete Long Tasks

#192

Post by scotty79 »

simonw wrote: post #18
eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

If you are not failing you are barely learning anything.

Re: Measuring AI Ability to Complete Long Tasks

#193

Post by dwohnitmok »

ehnto wrote: post #182
dwohnitmok wrote: post #180

Earlier quoted context omitted.

It's significantly accelerated to 4 months since the beginning of 2025, which puts 1 week within reach if things stay on trend. But yes 7 months is the more reliable long-term trend.

Can we attribute the acceleration to something specific, that might not actually continue growth? For example, agentic coding and reasoning models seem to have made a huge leap in abilities, but wouldn't translate to an ongoing exponential growth.

There's a fair amount of uncertainty on this point. In general it's unclear when/whether things will plateau out (although there are indications again that the trend is accelerating not decelerating).

That being said, if by "agentic coding" you are implying that a leap in capabilities is due to novel agentic frameworks/scaffolding that have appeared in 2025, I believe you are confusing cause and effect.

In particular, the agentic frameworks and scaffolding are by and large not responsible for the jump in capabilities. It is rather that the underlying models have improved sufficiently such that these frameworks and scaffolding work. None of the frameworks and scaffolding approaches of 2025 are new. All of them had been tried as early as 2023 (and indeed most of them had been tried in 2020 when GPT-3 came out). It's just that 2023-era models such as GPT-4 were far too weak to support them. Only in 2025 have models become sufficiently powerful to support these workflows.

Hence agentic frameworks and scaffolding are symptoms of ongoing exponential growth, not one-time boosts of growth.

Likewise reasoning models do not seem to be a one-time boost of growth. In particular reasoning models (or more accurate RLVR) seem to be an on-going source of new pretraining data (where the reasoning traces of models created during the process of RLVR serve as pretraining data for the next generation of models).

I remain uncertain, but I think there is a very real chance (>= 50%) that we are on an exponential curve that doesn't top out anytime soon (which gets really crazy really fast). If you want to do something about it, whether that's stopping the curve, flattening the curve, preparing yourself for the curve etc., you better do it now.

Re: Measuring AI Ability to Complete Long Tasks

#194

Post by simonw »

beasthacker wrote: post #191
simonw wrote: post #176

Earlier quoted context omitted.

If I had four hours to dedicate to this particular learning project I would still use LLMs to help me along the way, with the expectation that I'd learn more from those four hours than I would if I'd spent the same amount of time deliberately not using LLMs to help me. We've been given a tool that lets us ask questions in human language and get back answers that are correct 90% of the time! And that remaining 10% mea…

Hypothetical for you: Learn more if you tried to figure it out yourself for 3 hours then used the LLM for the last hour to unblock/check your work? Or learn more by utilizing LLM for help the whole four hours? My own experience is what I learn from an LLM sticks better if I take the former approach.

Depends on the task and my goals. If it was something new to me that I wanted to learn really deeply - and I had the four hours to spend - I might try the LLM-free route for the first three hours like you suggest.

If I found myself needing to do anything unrelated to the learning task, like knock out a quick Bash script, I'd still call on the LLM to get me out of that and help me stay focused on the new skill though.

Re: Measuring AI Ability to Complete Long Tasks

#195

Post by rishabhaiover »

dwohnitmok wrote: post #193
ehnto wrote: post #182

Earlier quoted context omitted.

Can we attribute the acceleration to something specific, that might not actually continue growth? For example, agentic coding and reasoning models seem to have made a huge leap in abilities, but wouldn't translate to an ongoing exponential growth.

There's a fair amount of uncertainty on this point. In general it's unclear when/whether things will plateau out (although there are indications again that the trend is accelerating not decelerating). That being said, if by "agentic coding" you are implying that a leap in capabilities is due to novel agentic frameworks/scaffolding that have appeared in 2025, I believe you are confusing cause and effect. In particular…

Well said. I don't think anybody's stopping anything. I wish I knew how to prepare for it.

Re: Measuring AI Ability to Complete Long Tasks

#196

Post by tacitusarc »

tacitusarc wrote: post #162
simonw wrote: post #73

Earlier quoted context omitted.

I've not had that problem at all with GPT-5.2 running in Codex CLI. I use prompts like this: Build a pure JavaScript library (no dependencies) for encoding and decoding this binary format. Start by looking at how the lite3-python library works - the JavaScript one should have the same API and probably the same code design too. Build the JS one in lite3-javascript - it should be a single JavaScript module which works…

I have not tried it in Codex CLI, I’ll give that a shot and see if it changes things.

It did make a noticeable difference
Post reply on HN