Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Measuring AI Ability to Complete Long Tasks

metr.org

Post reply on HN
Print view View original

171–180 of 196 posts

Re: Measuring AI Ability to Complete Long Tasks

#171

Post by girvo »

simonw wrote: post #112
girvo wrote: post #110

Earlier quoted context omitted.

> Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works I can only speak for myself, but the only way I've been able to learn things rapidly in this industry is by writing things myself: even rote re-typing of books or SO answers was enough to trigger this fo…

That's why I said "tweaking it yourself" - that's the point where you go beyond "just querying models and reading output".

That hasn't been enough for me, so far in my experience. I think I'm too crusty and set in my ways after 25 years of learning programming.

I have found them useful for general explanations and it's decent at finding me sources or directly answering questions about codebases/architecture (like where the @appendNode declarative mutation directive is fired in sequence for the Relay updater system), but it's code output I've not found a good teach tool for myself.

Re: Measuring AI Ability to Complete Long Tasks

#172

Post by mbbutler »

politelemon wrote: post #94
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

That's assuming everyone learns the same way, which isn't true. Watching a streamer beat a dark souls boss won't automatically make you competent at the game. Reading through gobs of code generated for you without knowing why various things were needed won't help either. A middle approach could be to get the LLM to guide you through the steps.

It's not just assuming that everyone learns the same way. It's assuming that everyone learns the way that all of the research literature on learning claims does not work.

Learning requires active recall/synthesis. Looking at solved examples instead of working them yourself does not suffice in math, physics, chemistry, or CS, but somehow it is supposed to work in this situation?

Re: Measuring AI Ability to Complete Long Tasks

#173

Post by beasthacker »

simonw wrote: post #18
eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

I buy the productivity argument, but I’m not convinced “30 minutes reading/tweaking agent output” is equivalent for learning to building it yourself.

If your goal is the feature, then yes: letting the agent do the heavy lifting and reviewing the diff afterward is a huge win.

But if your goal is understanding / skill-building, the hard part usually isn’t seeing a working solution. It’s doing the messy work of (a) making design choices, (b) getting stuck, (c) debugging, and (d) forming the mental model that lets you reproduce it later. Reviewing a correct implementation can create a feeling of “I get it,” but that feeling often doesn’t survive a blank file.

I’ve noticed this in my own hobby coding: LLMs are great for familiarity and unblocking progress, but the learning “sticks” much more when I’ve had to struggle through the failure modes myself. I’m watching the same dynamic play out with my son using ChatGPT to study for physics/calculus . . . it feels deep for him in the moment with the LLM, but exam-style transfer exposes the gaps.

Re: Measuring AI Ability to Complete Long Tasks

#174

Post by eleventen »

ModernMech wrote: post #134
eleventen wrote: post #16

Earlier quoted context omitted.

Respectfully, I think I’m in a better position to decide a) what value this has to me and b) what I choose to learn vs just letting Opus deal with. You don’t have enough information to say if I’ve saved time because you don’t know what I’m doing or what my goals are.

Respectfully, a) I didn't say anything about what value this has to you but moreover... b) you also don't have enough information to say if it's saved you time because the costs you will bear are in the future. Systems require maintenance, that's a fact you can't get rid of with AI. And often times, maintaining systems require more work than building them in the first place. Maintaining systems tends to require a dee…

> And you expect that this whole process will have saved you time, while you play games?

The topic in question is “Can AI tools do a task that would take a human 4 hours”. Not whether it can do that in a way that leads to maintainability or sustained learning. I’m noodling on a hobby project as leisure time. I got what I wanted. I had fun.

> incur future costs for not understanding it as it fails to act as you expect

That is your stronger argument. I’ve seen quality problems with the search results that come from using a smaller embedding model than I should. I don’t know yet if that’s a blocker or tolerable.

But I think that argument would be wrong too, because I’m very glad I chose Claude. The biggest limitation might be that I don’t have the compute locally to run an embedding model good enough to achieve decent results. It would have been a huge waste of my time to build it by hand and discover that at the end. I’m not about to pay for a sass vector DB or run this in AWS. At that level of effort I’d just scrap it.

Re: Measuring AI Ability to Complete Long Tasks

#175

Post by JohnnyMarcone »

scellus wrote: post #88
Davidzheng wrote: post #79

Big error bars and METR people are saying the longer end of the benchmark are less accurate right now. I think they mean this is a lower bound!

It's complicated. Opus 4.5 is actually not that good at the 80% threshold but is above others at 50% threshold of completion. I read there's a single task around 16h that the model completed, and the broad CI comes from that. METR currently simply runs out of tasks at 10-20h, and as a result you have a small N and lots of uncertainty there. (They fit a logistic to the discrete 0/1 results to get the thresholds you se…

Thanks for this comment. I've been trying to find anything about the huge error bars. Do you have any sources you can share for further reading?

Re: Measuring AI Ability to Complete Long Tasks

#176

Post by simonw »

beasthacker wrote: post #173
simonw wrote: post #18

Earlier quoted context omitted.

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

I buy the productivity argument, but I’m not convinced “30 minutes reading/tweaking agent output” is equivalent for learning to building it yourself. If your goal is the feature, then yes: letting the agent do the heavy lifting and reviewing the diff afterward is a huge win. But if your goal is understanding / skill-building, the hard part usually isn’t seeing a working solution. It’s doing the messy work of (a) maki…

If I had four hours to dedicate to this particular learning project I would still use LLMs to help me along the way, with the expectation that I'd learn more from those four hours than I would if I'd spent the same amount of time deliberately not using LLMs to help me.

We've been given a tool that lets us ask questions in human language and get back answers that are correct 90% of the time! And that remaining 10% means we have to engage critically with those answers, which is a useful learning trick in its own right.

Re: Measuring AI Ability to Complete Long Tasks

#177

Post by theptip »

rishabhaiover wrote: post #111
ehnto wrote: post #30

Earlier quoted context omitted.

I think you might be misunderstanding the article actually, this is about AI solving tasks as measured by how long it takes a human to solve the task. The AI could potentially solve it much quicker, but the use of "human time to solve" is an attempt to create a metric that reveals long horizon complexity (as I understand it anyway). It's interesting because like the article notes, AI is really smashing benchmarks, bu…

I've practiced a healthy skepticism of the recent boom but I can't reason why the long horizon time wouldn't stretch to 8 hours or a week worth's of effort from next year. After Opus-4.5, governments and organizations should really figure out a path out of this storm because we're in it now.

Doubling time has been 7 months for a while, so you should expect 8h not 1 week next year.

Re: Measuring AI Ability to Complete Long Tasks

#178

Post by throw1235435 »

simonw wrote: post #20
tokioyoyo wrote: post #14

Earlier quoted context omitted.

The point of eventual “all-code-is-written-by-AI” is that it really does not matter if your code is maintainable or not. In the end, most of the products are written to accomplish some sort of a goal or serve a need within a given set of restrictions (cost, speed and etc.). If the goal is achieved within given restrictions, the codebase can be thrown away until the next need is there to just create everything from sc…

I don't buy it. I think that could work , but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature. Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them. I expect that we'll learn over time that LLM-managed big…

I think the models, if they continue to get better, and frameworks/service patterns change to accommodate AI's. Where pieces of code will be thrown away, etc because the new code will be designed to slowly accommodate the "big ball of mud" risk.

We are moving from a conceptual/model job which typically requires training and skills (i.e. the code model/tool use/etc meets the requirements) to simply validation which is an easier problem and/or can be sharded in other roles. In other words the engineering part (i.e. the fun part) will be left to the AI. What I've found is people types (e.g. managers), and QA types (if it works, I don't care, this is what needs to work) will do well. People who liked the craftsman ship, solving problems, etc will do worse. Pure tech IMO will be less and less of a career.

Re: Measuring AI Ability to Complete Long Tasks

#179

Post by throw1235435 »

DANmode wrote: post #158
weitendorf wrote: post #31

Earlier quoted context omitted.

Generally I agree with your takes and find them very reasonable but in this case I think your deep experience might be coloring your views a bit. LLMs can hurt less experienced engineers by keeping them from building an intuition for why things work a certain way, or why an alternative won't work (or conversely, why an unconventional approach might not only be possible, but very useful and valuable!). I think problem…

What would you have us do, though? Stifle the tools, somehow? You’ve had nontechnical devs since npm, or before! No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.

> No: people that care to understand the whole stack, and be able to provide that value, will still exist and shine.

I hope so. But I don't believe so. I think us SWE's will find a way to disrupt that too as we all rush for the exits before this industry sinks.

The biggest barrier previously to anything (not just SWE) was the fact that like everything worth it in life; it takes work to see results. Generally people are time/resource poor and have to spend their own time or outsource the effort to get something - which limits the things they can do.

AI takes that away for SWE relative to other fields. People can get instant gratification now and "do it themselves" and given cost/benefit will prefer other fields now and want to spend their time elsewhere. At scale there will still be jobs for things people don't want to manage themselves but they will be more routine and busy work - not high salary skills based.

Re: Measuring AI Ability to Complete Long Tasks

#180

Post by dwohnitmok »

theptip wrote: post #177
rishabhaiover wrote: post #111

Earlier quoted context omitted.

I've practiced a healthy skepticism of the recent boom but I can't reason why the long horizon time wouldn't stretch to 8 hours or a week worth's of effort from next year. After Opus-4.5, governments and organizations should really figure out a path out of this storm because we're in it now.

Doubling time has been 7 months for a while, so you should expect 8h not 1 week next year.

It's significantly accelerated to 4 months since the beginning of 2025, which puts 1 week within reach if things stay on trend. But yes 7 months is the more reliable long-term trend.
Post reply on HN