Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two things LLM coding agents are still bad at

kix.dev

Post reply on HN
Print view View original

221–230 of 382 posts

Re: Two things LLM coding agents are still bad at

#221

Post by strangescript »

You don't want your agents to ask questions. You are thinking too short term. Its not ideal now, but agents that have to ask frequent questions are useless when it comes the vision of totally autonomous coding.

Humans ask questions of groups to fix our own personal short comings. It make no sense to try and master an internal system I rarely use, I should instead ask someone that maintains it. AI will not have this problem provided we create paths of observability for them. It doesn't take a lot of "effort" for them to completely digest an alien system they need to use.

Re: Two things LLM coding agents are still bad at

#222

Post by Romario77 »

dkarl wrote: post #208
rossant wrote: post #37

Recently, I asked Codex CLI to refactor some HTML files. It didn't literally copy and pasted snippets here and there as I would have done myself, it rewrote them from memory, removing comments in the process. There was a section with 40 successive links with complex URLs. A few days later, just before deployment to production, I wanted to double check all 40 links. First one worked. Second one worked. Third one worke…

I've had similar experience both in coding and in non-coding research questions. An LLM will do the first N right and fake its work on the rest. It even happens when asking an LLM to reformat a document, or asking it to do extra research to validate information. For example, before a recent trip to another city, I asked Gemini to prepare a list of brewery taprooms with certain information, and I discovered it had inc…

LLMs are not good at "cycles" - when you have to go over a list and do the same action on each item.

It's like it has ADHD and forgets or gets distracted in the middle.

And the reason for that is that LLMs don't have memory and process the tokens, so as they keep going over the list the context becomes bigger with more irrelevant information and they can lose the reason they are doing what they are doing.

Re: Two things LLM coding agents are still bad at

#223

Post by justonceokay »

strangescript wrote: post #221

You don't want your agents to ask questions. You are thinking too short term. Its not ideal now, but agents that have to ask frequent questions are useless when it comes the vision of totally autonomous coding. Humans ask questions of groups to fix our own personal short comings. It make no sense to try and master an internal system I rarely use, I should instead ask someone that maintains it. AI will not have this p…

If you look at a piece of architecture, you might be able to infer the intentions of the architect. However, there are many interpretations possible. So if you were to add an addendum to the building it makes sense that you might want to ask about the intentions.

I do not believe that AI will magically overcome the Chesterton Fence problem in a 100% autonomous way.

Re: Two things LLM coding agents are still bad at

#224

Post by joshribakoff »

My human fixed a bug by introducing a new one. Classic. Meanwhile, I write the lint rules, build the analyzers, and fix 500 errors before they’ve finished reading Stack Overflow. Just don’t ask me to reason about their legacy code — I’m synthetic, not insane.

—

Just because this new contributor is forced to effectively “SSH” into your codebase and edit not even with vim but with with sed and awk does not mean that this contributor is incapable of using other tools if empowered to do so. The fact it is able to work within such constraints goes to show how much potential there is. It is already much better at a human than erasing the text and re-typing it from memory and while it is a valid criticism that it needs to be taught how to move files imagine what it is capable of once it starts to use tools effectively.

—

Recently, I observed LLMs flail around for hours trying to get our e2e tests running as it tried to coordinate three different processes in three different terminals. It kept running commands in one terminal try to kill or check if the port is being used in the other terminal.

However, once I prompted the LLM to create a script for running all three processes concurrently, it is able to create that script, leverage it, and autonomously debug the tests now way faster than I am able to. It has also saved any new human who tries to contribute from similar hours of flailing around. Is there something we could have easily done by hand but just never had the time to do before LLMs. If anything, the LLM is just highlighting the existing problem in our codebase that some of us got too used to.

So yes, LLMs makes stupid mistakes, but so do humans, the thing is that LLms can ifentify and fix them faster (and better, with proper steering)

Re: Two things LLM coding agents are still bad at

#225

Post by wat10000 »

doikor wrote: post #55
ivape wrote: post #53

Earlier quoted context omitted.

You’re just not using LLMs enough. You can never trust the LLM to generate a url, and this was known over two years ago. It takes one token hallucination to fuck up a url. It’s very good at a fuzzy great answer, not a precise one. You have to really use this thing all the time and pick up on stuff like that.

I would generalise it to you can’t trust LLMs to generate any kind of unique identifier. Sooner or later it will hallucinate a fake one.

I would generalize it further: you can't trust LLMs.

They're useful, but you must verify anything you get from them.

Re: Two things LLM coding agents are still bad at

#226

Post by tjansen »

bunderbunder wrote: post #209
tjansen wrote: post #18

Agreed with the points in that article, but IMHO the no 1 issue is that agents only see a fraction of the code repository. They don't know whether there is a helper function they could use, so they re-implement it. When contributing to UIs, they can't check the whole UI to identify common design patterns, so they re-invent it. The most important task for the human using the agent is to provide the right context. "Loo…

This is what I keep running into. Earlier this week I did a code review of about new lines of code, written using Cursor, to implement a feature from scratch, and I'd say maybe 200 of those lines were really necessary. But, y'know what? I approved it. Because hunting down the existing functions it should have used in our utility library would have taken me all day. 5 years ago I would have taken the time because a PR…

>>because that's how management decided we should work, there's no point

If you are personally invested, there would be a point. At least if you plan to maintain that code for a few more years.

Let's say you have a common CSS file, where you define .warning {color: red}. If you want the LLM to put out a warning and you just tell it to make it red, without pointing out that there is the .warning class, it will likely create a new CSS def for that element (or even inline it - the latest Claude Code has a tendency to do that). That's fine and will make management happy for now.

But if later management decides that it wants all warning messages to be pink, it may be quite a challenge to catch every place without missing one.

Re: Two things LLM coding agents are still bad at

#227

Post by buildbot »

goalieca wrote: post #190
IanCal wrote: post #167

Earlier quoted context omitted.

They're moderately unreliable text copying machines if you need exact copying of long arbitrary strings. If that's what you want, don't use LLMs. I don't think they were ever really sold as that, and we have better tools for that. On the other hand, I've had them easily build useful code, answer questions and debug issues complex enough to escape good engineers for at least several hours. Depends what you want. They'…

> I don't think they were ever really sold as that, and we have better tools for that. We have OpenAI calling gpt5 as having PhD level of intelligence and others like Anthropoc saying it will write all our code within months. Some are claiming it’s already writing 70%. I say they are being sold as a magical do everything tool.

Would you hire a PhD to copy URLs by hand? Would them having PhD make it less likely they’d make a mistake than an high school student doing the same?

Re: Two things LLM coding agents are still bad at

#228

Post by tjansen »

hwillis wrote: post #148
tjansen wrote: post #18

Agreed with the points in that article, but IMHO the no 1 issue is that agents only see a fraction of the code repository. They don't know whether there is a helper function they could use, so they re-implement it. When contributing to UIs, they can't check the whole UI to identify common design patterns, so they re-invent it. The most important task for the human using the agent is to provide the right context. "Loo…

That's what claude.md etc are for. If you want it to follow your norms then you have to document them.

That's fine for norms, but I don't think you can use it to describe every single piece of your code. Every function, every type, every CSS class...

Re: Two things LLM coding agents are still bad at

#230

Post by stuartjohnson12 »

troupo wrote: post #214
alzoid wrote: post #179

Earlier quoted context omitted.

I asked Claude to add a debug endpoint to my hardware device that just gave memory information. It wrote 2600 lines of C that gave information about every single aspect of the system. On the one hand kind of cool. It looked at the MQTT code and the update code, the platform (esp) and generated all kinds of code. It recommended platform settings that could enable more detailed information that checked out when I looke…

> About half of the code generated fake data rather than actually do anything with the system. All the time // fake data. in production this would be real data ... proceeds to write sometimes hundreds of lines of code to provide fake data

"hey claude, please remove the fake data and use the real data"

"sure thing, I'll add logic to check if the real data exists and only use the fake data as a fallback in case the real data doesn't exist"

Post reply on HN