Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two things LLM coding agents are still bad at

kix.dev

Post reply on HN
Print view View original

161–170 of 382 posts

Re: Two things LLM coding agents are still bad at

#161

Post by thunky »

lenkite wrote: post #123
worldsayshi wrote: post #69

Earlier quoted context omitted.

This is of course bad but: humans also makes (different) mistakes all the time. We could account for the risk of mistakes being introduced and make more tools that validate things for us. In a way LLM:s encourage us to do this by adding other vectors of chaos into our work. Like, why not have tools built into our environment that checks that links are not broken? With the right architecture we could have validations…

In the above kind of described situation, a meticulous coder actually makes no mistakes. They will however make a LOT more mistakes if they use LLM's to do the same. I have already had to correct a LOT of crap similar to the above in refactoring-done-via-LLM over the last year. When stuff like this was done by a plain, slow, organic human, it was far more accurate. And many times, completely accurate with no defects.…

> I have already had to correct a LOT of crap similar to the above in refactoring-done-via-LLM over the last year

The person using the LLM should be reviewing their code before submitting it to you for review. If you can catch a copy paste error like this, then so should they.

The failure you're describing is that your coworkers are not doing their job.

And if you accept "the LLM did that, not me" as an excuse then the failure is on you and it will keep happening.

Re: Two things LLM coding agents are still bad at

#162

Post by IanCal »

hansmayer wrote: post #89
ivape wrote: post #53

Earlier quoted context omitted.

You’re just not using LLMs enough. You can never trust the LLM to generate a url, and this was known over two years ago. It takes one token hallucination to fuck up a url. It’s very good at a fuzzy great answer, not a precise one. You have to really use this thing all the time and pick up on stuff like that.

Yeah so, the reason people use various tools and machines in the first place is to simplify the work or everydays tasks by : 1) Making the tasks execute faster 2) Getting more reliable outputs then doing this by yourself 3) Making it repeatable . The LLMs obviously dont check any of these boxes so why don´t we stop pretending that we as users are stupid and don´t know how to use them and start taking them for what th…

They easily check a bunch of those boxes.

> why don´t we stop pretending that we as users are stupid and don´t know how to use them

This is in response to someone who saw a bunch of URLs coming out of it and was surprised at a bunch of them being wrong. That's using the tool wrong. It's like being surprised that the top results in google/app store/play store aren't necessarily the best match for your query but actually adverts!

Re: Two things LLM coding agents are still bad at

#163

Post by Ezhik »

tuesdaynight wrote: post #155
cimi_ wrote: post #112

Earlier quoted context omitted.

Speaking of agents and tests, here's a fun one I had the other day: while refactoring a large code base I told the agent to do something precise to a specific module, refactor with the new change, then ensure the tests are passing. The test suite is slow and has many moving parts; the tests I asked it to run take ~5 minutes. The thing decided to kill the test run, then it made up another command it said was the 'test…

I think that it's something that model providers don't want to fix, because the amount of times that Claude Code just decided to delete tests that were not passing before I added a memory saying that it would need to ask for my permission to do that was staggering. It stopped happening after the memory, so I believe that it could be easily fixed by a system prompt.

Your Claude Code actually respects CLAUDE.md?

Re: Two things LLM coding agents are still bad at

#164

Post by IanCal »

jollyllama wrote: post #156
ivape wrote: post #53

Earlier quoted context omitted.

You’re just not using LLMs enough. You can never trust the LLM to generate a url, and this was known over two years ago. It takes one token hallucination to fuck up a url. It’s very good at a fuzzy great answer, not a precise one. You have to really use this thing all the time and pick up on stuff like that.

> You’re just not using LLMs enough. > You can never trust the LLM to generate a url This is very poorly worded. Using LLMs more wouldn't solve the problem. What you're really saying is that the GP is uninformed about LLMs. This may seem like pedantry on my part but I'm sick of hearing "you're doing it wrong" when the real answer is "this tool can't do that." The former is categorically different than the latter.

It's pretty clearly worded to me, they don't use LLMs enough to know how to use them successfully. If you use them regularly you wouldn't see a set of urls without thinking "Unless these are extremely obvious links to major sites, I will assume each is definitely wrong".

> I'm sick of hearing "you're doing it wrong"

That's not what they said. They didn't say to use LLMs more for this problem. The only people that should take the wrong meaning from this are ones who didn't read past the first sentence.

> when the real answer is "this tool can't do that."

That is what they said.

Re: Two things LLM coding agents are still bad at

#165

Post by __MatrixMan__ »

Xss3 wrote: post #124
rossant wrote: post #37

Recently, I asked Codex CLI to refactor some HTML files. It didn't literally copy and pasted snippets here and there as I would have done myself, it rewrote them from memory, removing comments in the process. There was a section with 40 successive links with complex URLs. A few days later, just before deployment to production, I wanted to double check all 40 links. First one worked. Second one worked. Third one worke…

This is a horror story about bad quality control practices, not the use of LLMs.

I have a project that I've leaned heavily on LLM help for which I consider to embody good quality control practices. I had to get pretty creative to pull it off: spent a lot of time working on this sync system so that I can import sanitized production data into the project for every table it touches (there are maybe 500 of these) and then there's a bunch of hackery related to ensuring I can still get good test coverage even when some of these flows are partially specified (since adding new ones proceeds in several separate steps).

If it was a project written by humans I'd say they were crazy for going so hard on testing.

The quality control practices you need for safely letting an LLM run amok aren't just good. They're extreme.

Re: Two things LLM coding agents are still bad at

#166

Post by worldsayshi »

lenkite wrote: post #123
worldsayshi wrote: post #69

Earlier quoted context omitted.

This is of course bad but: humans also makes (different) mistakes all the time. We could account for the risk of mistakes being introduced and make more tools that validate things for us. In a way LLM:s encourage us to do this by adding other vectors of chaos into our work. Like, why not have tools built into our environment that checks that links are not broken? With the right architecture we could have validations…

In the above kind of described situation, a meticulous coder actually makes no mistakes. They will however make a LOT more mistakes if they use LLM's to do the same. I have already had to correct a LOT of crap similar to the above in refactoring-done-via-LLM over the last year. When stuff like this was done by a plain, slow, organic human, it was far more accurate. And many times, completely accurate with no defects.…

I think it goes without saying that we need to be sceptical when to use and not use LLM. The point I'm trying to make is more that we should have more validations and not that we should be less sceptical about LLMs.

Meticulousness shouldn't be an excuse to not have layers of validation that doesn't have to cost that much if done well.

Re: Two things LLM coding agents are still bad at

#167

Post by IanCal »

grey-area wrote: post #59
ivape wrote: post #53

Earlier quoted context omitted.

You’re just not using LLMs enough. You can never trust the LLM to generate a url, and this was known over two years ago. It takes one token hallucination to fuck up a url. It’s very good at a fuzzy great answer, not a precise one. You have to really use this thing all the time and pick up on stuff like that.

Or just not bother. It sounds pretty useless if it flunks on basic tasks like this. Perhaps you’ve been sold a lie?

They're moderately unreliable text copying machines if you need exact copying of long arbitrary strings. If that's what you want, don't use LLMs. I don't think they were ever really sold as that, and we have better tools for that.

On the other hand, I've had them easily build useful code, answer questions and debug issues complex enough to escape good engineers for at least several hours.

Depends what you want. They're also bad (for computers) at complex arithmetic off the bat, but then again we have calculators.

Re: Two things LLM coding agents are still bad at

#168

Post by imcritic »

About the first point mentioned in article: could that problem be solved simply by changing the task from something like "refactor this code" to something like "refactor this code as a series of smaller atomic changes (like moving blocks of code or renaming variable references in all places), disable suitable for git commits (and provide git message texts for those commits)"?

Re: Two things LLM coding agents are still bad at

#169

Post by AllegedAlec »

coldtea wrote: post #125
AllegedAlec wrote: post #114

Earlier quoted context omitted.

That would be a solution, yes. But currently it feels extremely borked from a UX perspective. It purports to be able to do this, but when you tell it to it breaks in unintuitive ways. I hate this idea of "well you just need to understand all the arcane ways in which to properly use it to its proper effects". It's like a car which has a gear shifter, but that's not fully functional yet, so instead you switch gear by s…

> But currently it feels extremely borked from a UX perspective. It purports to be able to do this, but when you tell it to it breaks in unintuitive ways. Thankfully as programmers we know better and don't need to care what the UI pretends to be able to do :) > We would rightfully stone any engineer who'd design this and then say "well obvious user error" when the user rightfully complains that they crash whenever th…

> >But currently it feels extremely borked from a UX perspective. It purports to be able to do this, but when you tell it to it breaks in unintuitive ways.

> Thankfully as programmers we know better and don't need to care what the UI pretends to be able to do :)

But we do though. You can't just say "yeah they left all the foot guns in but we ought to know not to use them", especially not when the industry shills tell you those footguns are actually rocket boosters to get you to the fucking moon and back.

Re: Two things LLM coding agents are still bad at

#170

Post by cheema33 »

Leynos wrote: post #106
cheema33 wrote: post #71

From the article: > I contest the idea that LLMs are replacing human devs... AI is not able to replace good devs. I am assuming that nobody sane is claiming such a thing today. But, it can probably replace bad and mediocre devs. Even today. In my org we had 3 devs who went through a 6-month code boot camp and got hired a few years ago when it was very difficult to find good devs. They struggled. I would give them eas…

What do you think was the reason that the bootcamp grads struggling to get better at what they do?

A computer science degree in most US colleges takes about 4 years of work. Boot camps try to cram that into 6 months. All the while many students have other full-time jobs. This is simply not enough training for the students to start solving complex real world problem. Even 4 years is not enough.

Many companies were willing to hire fresh college grads in the hopes that they could solve relatively easy problems for a few years, gain experience and become successful senior devs at some point.

However, with the advent of AI dev tools, we are seeing very clear signs that junior dev hiring rates have fallen off a cliff. Our project manager, who has no dev experience, frequently assigns easy tasks/github issues to Github Copilot. Copilot generates a PR in a few minutes that other devs can review before merging. These PRs are far superior to what an average graduate of a code boot camp could ever create. Any need we had for a junior dev has completely disappeared.

Post reply on HN