Measuring AI Ability to Complete Long Tasks • Hacker News Board

Re: Measuring AI Ability to Complete Long Tasks

#101

Post by Applejinx » Sun, Dec 21, 2025, 12:37 PM UTC

simonw wrote: post #18

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

This really makes for a good natural experiment: carry on :)

I have a hard time imagining how much you'd have to literally bribe me to get me to try doing it the way you describe. I'm too interested in implementation details of things and looking for innovations—in fact I make my living doing that, like some cyberpunk gremlin just delighting in messing with stuff in unexpected ways. I don't understand why you're not, but maybe it's not for me to understand.

Carry on. We'll check back and see how it worked for ya :)

Re: Measuring AI Ability to Complete Long Tasks

#102

Post by fluidcruft » Sun, Dec 21, 2025, 12:42 PM UTC

Avicebron wrote: post #45

fluidcruft wrote: post #38

Earlier quoted context omitted.

I sort of disagree. It's somewhat like having hypercard again. You can build fun UI things and make machines do what you want them to do. You can care about the parts you want to care about and not sweat about the parts you don't want to learn in detail (yet). And Claude and codex make great guides/Sherpas. There are just too many parts involved to do anything. For example today I built a simple data collection app t…

>only wasted 5min of Claude's time rather than hours of mine I mean will you (we) retain all that it did after a few months go by? You may say we don't need to, but that sounds a little shallow given we're both on HN. Do you remember Gatsby's criticism of "Summer People"?

I don't even remember things I did two years ago unless I leave good breadcrumbs and documentation. I don't think it's particularly worse than pulling in some dependency or framework from GitHub that will be completely different next year anyway. And Google's prone to change anything in Android anyway. Mobile or web seems like a foundation of quicksand, it's not anything I care about. The real takeaway is I can be productive without wasting my time on all the damn churn by just-in-time learning aided by these tools.

I'm pretty sure I will remember how easy and correct it was to modify Caddy vs the months of putzing around building Rube Goldberg constellations of services crap that I did last year for a different thing and that even Claude wanted to do. I've done the whole wading through outdated blog posts and trying to read documentation on other projects that I was doing. Learning five different projects and having to maintain seven services running in docker just so that I can use a tool to capture photos and store them in a webapp that can only be used if connected to my own VPN is insane and it's why I am not a web developer. I will 100% remember what 200 lines of golang does after looking at it again. The 1000 lines of JavaScript that were and backend auth crap that no longer exists: good riddance.

And no I don't trust my memory about what Gatsby said about Summer people without looking it up. I read Gatsby 30 years ago.

Re: Measuring AI Ability to Complete Long Tasks

#103

Post by Applejinx » Sun, Dec 21, 2025, 12:43 PM UTC

extr wrote: post #61

johnfn wrote: post #46

Earlier quoted context omitted.

But how much of that time is truly spent on learning relevant knowledge, and how much of it is just (now) useless errata? Take vector search for an example. Pre-GPT, I would spend like an hour chasing down a typo, like specifying 1023 instead of 1024 or something. This sort of problem is now trivially solved in minutes by a LLM that fully understands the API surface area. So what exactly do I lose by not spending tha…

I think people fool themselves with this kind of thing a lot. You debug some issue with your GH actions yaml file for 45 minutes and think you "learned something", but when are you going to run into that specific gotcha again? In reality the only lasting lesson is "sometimes these kinds of yaml files can be finnicky". Which you probably already knew at the outset. There's no personal development in continually bashin…

Github Actions, web development, stuff like that, are terrible examples of where not to use AI.

You can't really go to giant piles of technical debt and look to those for places to be human. It's soul-destroying. My concern would be that vibe coding will make those places of soul-less technical debt even deeper and deadlier. There will be nobody there, for generations of cruft. Where once the technical debt was made by committee, now it'll be the ghosts of committees, stirred up by random temperature, only to surface bits of rot that just sink down into the morass again, unfixed.

When 'finicky' is actually an interesting problem, or a challenge, that's one thing. When 'finicky' is just 'twelve committees re-hacked this and then it's been maintained by LLMs for years', there is nothing gained by trying to be human at it.

Re: Measuring AI Ability to Complete Long Tasks

#104

Post by risyachka » Sun, Dec 21, 2025, 12:46 PM UTC

simonw wrote: post #18

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

Reading without actually doing does not really result in learning, only very marginal one.

Try reading tutorials on a new programming language for 30 minutes and then open new text file and write basic loop with print.

It won’t even compile- which shows you haven’t really learned anything. Just read an interesting story. Sure you pita few bits here and there but you still don’t know how to do even the moat basic thing.

Re: Measuring AI Ability to Complete Long Tasks

#105

Post by csomar » Sun, Dec 21, 2025, 1:02 PM UTC

simonw wrote: post #18

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

The struggle is how you learn. I think that’s pretty much established scientifically by now?

Re: Measuring AI Ability to Complete Long Tasks

#106

Post by Leynos » Sun, Dec 21, 2025, 1:03 PM UTC

Aperocky wrote: post #24

I think the problem here is LLM eventually pollute its context window with so much of the current task that the larger picture or architectural sanity is forgotten in favor of the current task at hand. And rarely is a software one and done, with a few round like this, the software architecture would have become schizophrenic. Combating this tendency usually require a lot of the work of these "long task" to be thrown…

This was why server-side compaction in GPT-5.2 was such a big deal. The model is by default provided with a tool that will prioritise the initial task and salient updates in context window compaction, and the new model has been trained to use it.

Re: Measuring AI Ability to Complete Long Tasks

#107

Post by fluidcruft » Sun, Dec 21, 2025, 1:34 PM UTC

tokioyoyo wrote: post #23

simonw wrote: post #20

Earlier quoted context omitted.

I don't buy it. I think that could work , but it can work in the same way that plenty of big companies have codebases that are a giant ball of mud and yet they somehow manage to stay in business and occasionally ship a new feature. Meanwhile their rivals with well constructed codebases who can promptly ship features that work are able to run rings around them. I expect that we'll learn over time that LLM-managed big…

Fair enough. In my imagination, I can see people writing AI-first framework/architectures and a general trend for people to “migrate to such frameworks”, just like the push towards the microservices architectures in 2010s. A part of these frameworks would be “re-constructibility” by changing contracts in parts where it matters, and somehow the framework would make it easy for the LLM to discover such “parts”. Honestl…

My experience with LLM and agents has led to the opinion that a LLM-friendly codebase is actually a very human friendly code base.

Re: Measuring AI Ability to Complete Long Tasks

#108

Post by 9rx » Sun, Dec 21, 2025, 1:38 PM UTC

dangus wrote: post #77

visarga wrote: post #68

Earlier quoted context omitted.

You should take into consideration the time it took to make those 9200 tests originally. If you have good test coverage the agent can go much farther ahead.

Heh, I mostly use AI in the opposite direction to write tests because: 1. That’s the part of development work I hate the most and never really clicked with me 2. AI to to this point seems to be better at writing tests than code Take this with the grain of salt that: 1. I suck 2. My work is mostly in the realm of infrastructure where testing has always been weird and a little dumb

AI has become very good at writing pointless and bad tests, at least. It remains difficult to compel it to write good tests consistently.

But even if it wrote great tests every time, the trouble is that testing was designed around the idea of "double entry accounting". Even great tests can test the wrong thing. In the old world you would write a test case and then implement something to satisfy the same. If both sides of the ledger agree, so to speak, you can be pretty confident that both are correct. — In other words, going through the process of implementation gives an opportunity to make sure the test you wrote isn't ill-conceived or broken itself. If you only write the tests, or only write the implementation, or write none of it, there is no point at which you can validate your work.

If you have already built up an application and are reusing its test suite to reimplement the software in another language, like above, that is one thing, but in greenfield work it remains an outstanding problem of how to validate the work when you start to involve AI agents. Another article posted here recently suggests that we can go back to manual testing to validate the work... But that seems like a non-solution.

Re: Measuring AI Ability to Complete Long Tasks

#109

Post by scotty79 » Sun, Dec 21, 2025, 1:51 PM UTC

> As shown above, when we fit a similar trend to just the 2024 and 2025 data, this shortens the estimate of when AI can complete month-long tasks with 50% reliability by about 2.5 years.

I don't think I have 50% success rate at month long tasks.

Anything that exceeds one day is pretty hard.

Re: Measuring AI Ability to Complete Long Tasks

#110

Post by girvo » Sun, Dec 21, 2025, 1:59 PM UTC

simonw wrote: post #18

eleventen wrote: post #5

I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…

I don't think building it the long way is necessarily a more effective way to learn. You could spend 4 hours (that you don't have) building that feature. Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works. My hunch is that the 30 minutes of focused learnin…

> Or... you could have the coding agent build it in the background for you in 15 minutes, then spend 30 minutes reading through what it did, tweaking it yourself and peppering it with questions about how it all works

I can only speak for myself, but the only way I've been able to learn things rapidly in this industry is by writing things myself: even rote re-typing of books or SO answers was enough to trigger this for me.

Just querying models and reading output doesn't seem to work for me, but that's maybe down to my particular learning style.