Two things LLM coding agents are still bad at • Hacker News Board

Re: Two things LLM coding agents are still bad at

#251

Post by goalieca » Thu, Oct 09, 2025, 3:25 PM UTC

buildbot wrote: post #227

goalieca wrote: post #190

Earlier quoted context omitted.

> I don't think they were ever really sold as that, and we have better tools for that. We have OpenAI calling gpt5 as having PhD level of intelligence and others like Anthropoc saying it will write all our code within months. Some are claiming it’s already writing 70%. I say they are being sold as a magical do everything tool.

Would you hire a PhD to copy URLs by hand? Would them having PhD make it less likely they’d make a mistake than an high school student doing the same?

Grad students and even post docs often do a lot of this manual labour for data entry and formatting. Been there, done that.

Re: Two things LLM coding agents are still bad at

#252

Post by parineum » Thu, Oct 09, 2025, 3:25 PM UTC

buildbot wrote: post #227

goalieca wrote: post #190

Earlier quoted context omitted.

> I don't think they were ever really sold as that, and we have better tools for that. We have OpenAI calling gpt5 as having PhD level of intelligence and others like Anthropoc saying it will write all our code within months. Some are claiming it’s already writing 70%. I say they are being sold as a magical do everything tool.

Would you hire a PhD to copy URLs by hand? Would them having PhD make it less likely they’d make a mistake than an high school student doing the same?

A high school student would use copy/paste and the urls would be perfect duplicates..

Re: Two things LLM coding agents are still bad at

#253

Post by weakfish » Thu, Oct 09, 2025, 3:26 PM UTC

stuartjohnson12 wrote: post #230

troupo wrote: post #214

Earlier quoted context omitted.

> About half of the code generated fake data rather than actually do anything with the system. All the time // fake data. in production this would be real data ... proceeds to write sometimes hundreds of lines of code to provide fake data

"hey claude, please remove the fake data and use the real data" "sure thing, I'll add logic to check if the real data exists and only use the fake data as a fallback in case the real data doesn't exist"

This comment captures exactly what aggravates me about CC / other agents in a way that I wasn't sure how to express before. Thanks!

Re: Two things LLM coding agents are still bad at

#254

Post by mbesto » Thu, Oct 09, 2025, 3:27 PM UTC

IanCal wrote: post #162

hansmayer wrote: post #89

Earlier quoted context omitted.

Yeah so, the reason people use various tools and machines in the first place is to simplify the work or everydays tasks by : 1) Making the tasks execute faster 2) Getting more reliable outputs then doing this by yourself 3) Making it repeatable . The LLMs obviously dont check any of these boxes so why don´t we stop pretending that we as users are stupid and don´t know how to use them and start taking them for what th…

They easily check a bunch of those boxes. > why don´t we stop pretending that we as users are stupid and don´t know how to use them This is in response to someone who saw a bunch of URLs coming out of it and was surprised at a bunch of them being wrong. That's using the tool wrong . It's like being surprised that the top results in google/app store/play store aren't necessarily the best match for your query but actua…

> This is in response to someone who saw a bunch of URLs coming out of it and was surprised at a bunch of them being wrong. That's using the tool wrong. It's like being surprised that the top results in google/app store/play store aren't necessarily the best match for your query but actually adverts!

The CEO of Anthropic said I can fire all of my developers soon. How could one possibly be using the tool wrong? /s

Re: Two things LLM coding agents are still bad at

#255

Post by indigodaddy » Thu, Oct 09, 2025, 3:27 PM UTC

rossant wrote: post #79

coldtea wrote: post #76

Earlier quoted context omitted.

> A few days later, just before deployment to production, I wanted to double check all 40 links. This was allowed to go to master without "git diff" after Codex was done?

It was a fairly big refactoring basically converting a working static HTML landing page into a Hugo website, splitting the HTML into multiple Hugo templates. I admit I was quite in a hurry and had to take shortcuts. I didn't have time to write automated tests and had to rely on manual tests for this single webpage. The diff was fairly big. It just didn't occur to me that the URLs would go through the LLMs and could b…

This is why my instinct for this sort of task is, "write a script that I can use to do x y z," instead of "do x y z"

Re: Two things LLM coding agents are still bad at

#256

Post by ojosilva » Thu, Oct 09, 2025, 3:32 PM UTC

majora2007 wrote: post #211

I think LLMs provide value, used it this morning to fix a bug in my PDF Metadata parser without having to get too deep into the PDF spec. But most of the time, I find that the outputs are nowhere near the effect of just doing it myself. I tried Codex Code the other day to write some unit tests. I had a few setup and wanted to use it (because mocking the data is a pain). It took about 8 attempts, I had to manually fix…

I think LLMs have what it takes at this point in time, but it's the coding agent (combined with the model) that make the magic happen. Coding agents can implement copy-pasting, it's a matter of building the right tool for it, then iterating with given models/providers, etc. And that's true for everything else that LLMs lack today. Shortcomings can be remediated with good memory and context engineering, safety-oriented instructions, endless verification and good overall coding agent architecture. Also having a model that can respond fast, have a large context window and maintain attention to instructions is also essential for a good overall experience.

And the human prompting, of course. It takes good sw engineering skills, particularly knowing how to instruct other devs in getting the work done, setting up good AGENTS.md (CLAUDE.md, etc) with codebase instructions, best practices, etc etc.

So it's not an "AI/LLMs are capable of replacing developers"... that's getting old fast. It's more like, paraphrasing the wise "it's not what your LLM can do for you, but what can you do for your LLM"

Re: Two things LLM coding agents are still bad at

#257

Post by mbesto » Thu, Oct 09, 2025, 3:33 PM UTC

IanCal wrote: post #248

goalieca wrote: post #190

Earlier quoted context omitted.

> I don't think they were ever really sold as that, and we have better tools for that. We have OpenAI calling gpt5 as having PhD level of intelligence and others like Anthropoc saying it will write all our code within months. Some are claiming it’s already writing 70%. I say they are being sold as a magical do everything tool.

Intelligence isn't the same as "can exactly replicate text". I'm hopefully smarter than a calculator but it's more reliable at maths than me. Also there's a huge gulf between "some people claim it can do X" and "it's useful". Altman promising something new doesn't decrease the usefulness of a model.

What you are describing is "dead reasoning zones".[0]

    "This isn't how humans work. Einstein never saw ARC grids, but he'd solve them instantly. Not because of prior knowledge, but because humans have consistent reasoning that transfers across domains. A logical economist becomes a logical programmer when they learn to code. They don't suddenly forget how to be consistent or deduce.

    But LLMs have "dead reasoning zones" — areas in their weights where logic doesn't work. Humans have dead knowledge zones (things we don't know), but not dead reasoning zones. Asking questions outside the training distribution is almost like an adversarial attack on the model."

https://jeremyberman.substack.com/p/how-i-got-the-highest-sc...

Re: Two things LLM coding agents are still bad at

#258

Post by hitarpetar » Thu, Oct 09, 2025, 3:37 PM UTC

IanCal wrote: post #162

hansmayer wrote: post #89

Earlier quoted context omitted.

Yeah so, the reason people use various tools and machines in the first place is to simplify the work or everydays tasks by : 1) Making the tasks execute faster 2) Getting more reliable outputs then doing this by yourself 3) Making it repeatable . The LLMs obviously dont check any of these boxes so why don´t we stop pretending that we as users are stupid and don´t know how to use them and start taking them for what th…

They easily check a bunch of those boxes. > why don´t we stop pretending that we as users are stupid and don´t know how to use them This is in response to someone who saw a bunch of URLs coming out of it and was surprised at a bunch of them being wrong. That's using the tool wrong . It's like being surprised that the top results in google/app store/play store aren't necessarily the best match for your query but actua…

it's amazing that you picked another dark pattern as your comparison

Re: Two things LLM coding agents are still bad at

#259

Post by weakfish » Thu, Oct 09, 2025, 3:37 PM UTC

username223 wrote: post #232

cheema33 wrote: post #170

Earlier quoted context omitted.

A computer science degree in most US colleges takes about 4 years of work. Boot camps try to cram that into 6 months. All the while many students have other full-time jobs. This is simply not enough training for the students to start solving complex real world problem. Even 4 years is not enough. Many companies were willing to hire fresh college grads in the hopes that they could solve relatively easy problems for a…

> Any need we had for a junior dev has completely disappeared. Where do your senior devs come from?

That's the question that has been stuck in my head as I read all these stories about junior dev jobs disappearing. I'm firmly mid-level, having started my career just before LLM coding took off. Sometimes it feels like I got on the last chopper out of Saigon.

Re: Two things LLM coding agents are still bad at

#260

Post by hitarpetar » Thu, Oct 09, 2025, 3:39 PM UTC

IanCal wrote: post #248

goalieca wrote: post #190

Earlier quoted context omitted.

> I don't think they were ever really sold as that, and we have better tools for that. We have OpenAI calling gpt5 as having PhD level of intelligence and others like Anthropoc saying it will write all our code within months. Some are claiming it’s already writing 70%. I say they are being sold as a magical do everything tool.

Intelligence isn't the same as "can exactly replicate text". I'm hopefully smarter than a calculator but it's more reliable at maths than me. Also there's a huge gulf between "some people claim it can do X" and "it's useful". Altman promising something new doesn't decrease the usefulness of a model.

saddest goalpost ever