Measuring AI Ability to Complete Long Tasks
1–10 of 196 posts
Re: Measuring AI Ability to Complete Long Tasks
#2This seems like a good way to measure LLM improvement.
It matches the my personal feeling when using progressively better models over time.
Re: Measuring AI Ability to Complete Long Tasks
#3Opus is already the name of an audio codec.
Re: Measuring AI Ability to Complete Long Tasks
#4Opus is already the name of an audio codec.
Opus: "an artistic work, especially one on a large scale."
The names Haiku, Sonnet, and Opus have not been chosen randomly.
Re: Measuring AI Ability to Complete Long Tasks
#5I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet.
I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders and worked on the first try.
Afterward, I sort of had to reflect on the fact that I learned essentially nothing about building vector search. I wanted the feature more than I wanted to know how to build the feature. It kept me learning the thing I cared about rather than doing a side quest.
Re: Measuring AI Ability to Complete Long Tasks
#6Opus is already the name of an audio codec.
Have you been living under a rock?
Re: Measuring AI Ability to Complete Long Tasks
#7Would be interesting to see Gemini 3.0 Pro benchmarked as well.
Re: Measuring AI Ability to Complete Long Tasks
#8I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…
The result of you having worked 4 hours to implement the thing is not just that you have the thing, it's that you have the thing and you understand the thing. Having the thing is next to useless if you don't understand it.
At best it plods along as you keep badgering Claude to fix it, until inevitably Claude reaches a point where it can't help. At which time you'll be forced to spend at least the 4 hours you would have originally spent trying to understand it so you can fix it yourself.
At worst the thing will actively break other things you do understand in ways you don't understand, and you'll have to spend at least 4 hours cleaning up the mess.
Either way it's not clear you've saved any time at all.
Re: Measuring AI Ability to Complete Long Tasks
#9[dead]
Re: Measuring AI Ability to Complete Long Tasks
#10I recently asked Opus to just “Add vector search” to my current hobby project, a topic I know very little about. It set up manticore, pulled an embedding model, wrote a migration tool for my old keyword indices, and built the front end. I’m not exaggerating much either: the prompt was the length of a tweet. I think it would easily have taken me 4+ hours to do that. It ran in 15 minutes while I played Kirby Air Riders…
Yeah and then it becomes an unmaintainable monolith because at some point the AI also lost track of what code does what.
Great for Opus because you’re now a captive customer.