My problem with the OpenAI models (GPT5.2 in particular) recently is an extreme aversion to doing more than the smallest step in a task before asking for using input. Even if I explicitly instruct it to continue without input until the task is complete, it ignores the instruction. I cannot imagine GPT5.2 working on a task for more than 2 minutes, let alone 4 hours. I’m curious if you’ve run into this and figured out…
What agent framework are you using? It can differ from one to the next on the same model.
My problem with the OpenAI models (GPT5.2 in particular) recently is an extreme aversion to doing more than the smallest step in a task before asking for using input. Even if I explicitly instruct it to continue without input until the task is complete, it ignores the instruction. I cannot imagine GPT5.2 working on a task for more than 2 minutes, let alone 4 hours. I’m curious if you’ve run into this and figured out…
I've not had that problem at all with GPT-5.2 running in Codex CLI. I use prompts like this: Build a pure JavaScript library (no dependencies) for encoding and decoding this binary format. Start by looking at how the lite3-python library works - the JavaScript one should have the same API and probably the same code design too. Build the JS one in lite3-javascript - it should be a single JavaScript module which works…
I have not tried it in Codex CLI, I’ll give that a shot and see if it changes things.
IMHO, in the software field, learning can be simpler to 2 phases. The first one is exploration, where we read blogs, docs, and books; listen to lectures and talks. Then comes the second phase of exploitation, where we actually use the thing we learned. You can think of all those “learning from scratch” videos as someone who is doing the phase 2. I love the phase one and most of the time don’t have time and energy to sit down and go through the phase 2. Nowadays, I feel like the 2 phases are combined, thanks to LLMs. For instance, I wanted to do some animation for visualizations. This week, I learned AnimeJS by watching CCAgent create the animation I wanted, which was interspersed with questions that were answered with diagrams and text, which accomplishes the phase 1. I do not like letting them run the show. Then comes phase 2, where I organize the code, abstract things, rewrite code, still use their help for long rewrites, but totally my ideas and mine only. This saves time tremendously.
I genuinely do think you can learn 90% of what that is to learn about integrating with a vector database from having an LLM do the work for you and then carefully reviewing what it did. Turns out there's science that backs me up here: https://en.wikipedia.org/wiki/Worked-example_effect - showing people "worked examples" can be more effective than making them solve the problem themselves. That Wikipedia article is a l…
> Worked examples are step-by-step illustrations of the process required to complete a task or solve a problem. That’s not what having a bot generate your integration is and reading it post-facto is. The bot isn’t guiding you through the process so you can go do it yourself. At best you would use this as a reference to go do another integration yourself - but at this point why even bother when you can just get the bo…
> The bot isn’t guiding you through the process so you can go do it yourself.
It is if you ask it to. Learning well with LLMs requires a lot of self-discipline - you have to be actively aware of the threat that you won't actually learn effectively and take steps to counter that.
I keep meticulous notes of everything these things do for me, which adds up to a valuable set of notes over time. I gave up on remembering things without notes a long time ago!
They were right. I hadn't read enough of the article to understand what was meant by multi-hour tasks. I upvoted them for pointing that out.
>> Maybe this is the negative effects of excessive LLM usage that are spoken about. > I upvoted them for pointing that out. I'm also curious about what you think about the GPs question. TBH, responding after reading half an article was a common thing for most people pre-LLM anyway.
Yeah, show me a Hacker News user who's never posted a comment on a story without properly reading it (or even without clicking the link). LLMs have nothing to do with it.
If I had piped the article through an LLM first, I wouldn't have made the embarrassing mistake in that comment!
Heh, I mostly use AI in the opposite direction to write tests because: 1. That’s the part of development work I hate the most and never really clicked with me 2. AI to to this point seems to be better at writing tests than code Take this with the grain of salt that: 1. I suck 2. My work is mostly in the realm of infrastructure where testing has always been weird and a little dumb
AI has become very good at writing pointless and bad tests, at least. It remains difficult to compel it to write good tests consistently. But even if it wrote great tests every time, the trouble is that testing was designed around the idea of "double entry accounting". Even great tests can test the wrong thing. In the old world you would write a test case and then implement something to satisfy the same. If both side…
Every error is a signal you need better tests. You can let the LLM create tests for every error it stumbles into, besides all the regular tests it can write on its own. Add all test scenarios you can think of, since you are not implementing them by hand. A bad test is invalidated by code, and a bad code invalidated by the test, so between them the AI agent can become reliable.
The key insight from this benchmark is using "human-equivalent hours" rather than actual AI execution time. It's measuring capability complexity, not speed.
What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed.
This is why I think the current "agent" paradigm needs human checkpoints at regular intervals. Let the AI work for 30 minutes, then review progress. Repeat. This way you catch drift early before it compounds.
The other thing missing from these benchmarks: recovery ability. When the AI gets stuck on hour 3 of a 4-hour task, can it recognize the problem and backtrack? Or does it confidently continue down the wrong path?
The key insight from this benchmark is using "human-equivalent hours" rather than actual AI execution time. It's measuring capability complexity, not speed. What's interesting is the 50% vs 80% reliability gap. At 50% success rate on a 4-hour task, you're essentially gambling. If it fails, you've potentially wasted the 4 hours plus the time debugging why it failed. This is why I think the current "agent" paradigm nee…
> This is why I think the current "agent" paradigm needs human checkpoints at regular intervals. Let the AI work for 30 minutes, then review progress. Repeat. This way you catch drift early before it compounds.
The problem with this approach is that in 30 minutes, an agent is able to produce a massive amount of stuff. Reviewing all this is a nightmare, in the sense that on the surface it seems fine and it often works, until it doesn't. The bugs introduced are often subtle and their effects manifest later, if ever.
So, for stuff that matters (to me), I prefer not to use agents at all.
Maybe things will change in a year, or 5, or 10. I will be giving it a try. but for the moment it's just not worth it, and the upside-down workflow it pushes on me is just making me tired and lose satisfaction from doing my job.