Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two things LLM coding agents are still bad at

kix.dev

Post reply on HN
Print view View original

341–350 of 382 posts

Re: Two things LLM coding agents are still bad at

#341

Post by kypro »

willsmith72 wrote: post #334
AlwaysRock wrote: post #330

Earlier quoted context omitted.

Asking for a source from llms is so eye opening. I am yet to have them link a source that actually supports what they said.

> I am yet to have them link a source that actually supports what they said. You're not trying very hard then. Here, my first try: https://claude.ai/share/ef7764d3-6c5c-4d1a-ba28-6d5218af16e0

But no one uses LLMs like this. This is the type of simple fact you could just Google and check yourself.

LLMs are useful for providing answers to more complex questions where some reasoning or integration of information is needed.

In these cases I mostly agree with the parent commenter. LLMs often come up with plausibly correct answers, then when you ask to cite sources they seem to just provide articles vaguely related to what they said. If you're lucky it might directly address what the LLM claimed.

I assume this is because what LLMs say is largely just made up, then when you ask for sources it has to retroactively try to find sources to justify what it said, and it often fails and just links something which could plausibly be a source to back up it's plausibly true claims.

Re: Two things LLM coding agents are still bad at

#342

Post by fwip »

steveklabnik wrote: post #296
radarsat1 wrote: post #268

Earlier quoted context omitted.

Agreed. You basically want an LLM to have a tool that writes its own agent to accomplish a repetitive task. I think this is doable.

You can already sort of do this by asking it to write a script to do the refactor. Claude sometimes suggests this on its own to me even. But obviously sometimes larger refactors aren't easy to implement in bash.

Right - and ideally, after writing the script to do the task, it could discard all the tokens involved in writing the script.

Re: Two things LLM coding agents are still bad at

#343

Post by schmichael »

kypro wrote: post #341
willsmith72 wrote: post #334

Earlier quoted context omitted.

> I am yet to have them link a source that actually supports what they said. You're not trying very hard then. Here, my first try: https://claude.ai/share/ef7764d3-6c5c-4d1a-ba28-6d5218af16e0

But no one uses LLMs like this. This is the type of simple fact you could just Google and check yourself. LLMs are useful for providing answers to more complex questions where some reasoning or integration of information is needed. In these cases I mostly agree with the parent commenter. LLMs often come up with plausibly correct answers, then when you ask to cite sources they seem to just provide articles vaguely rel…

I do, and so does Google. When I googled "When was John Howard elected?" the correct answer came back faster in the AI Overview than I could find the answer in the results. The source the AI Overview links even provides confirmation of the correct answer.

Re: Two things LLM coding agents are still bad at

#345

Post by worldsayshi »

rullelito wrote: post #74
worldsayshi wrote: post #69

Earlier quoted context omitted.

This is of course bad but: humans also makes (different) mistakes all the time. We could account for the risk of mistakes being introduced and make more tools that validate things for us. In a way LLM:s encourage us to do this by adding other vectors of chaos into our work. Like, why not have tools built into our environment that checks that links are not broken? With the right architecture we could have validations…

LLMs are turning into LLMs+hard-coded fixes for every imaginable problem.

Why hard coded?

Re: Two things LLM coding agents are still bad at

#346

Post by paulhebert »

purplerabbit wrote: post #340
tzs wrote: post #272

Just the other day I hit something that I hadn't realized could happen. It was not code related in my case, but could happen with code or code-related things (and did to a coworker). In a discussion here on HN about why a regulation passed 15 years ago was not as general as it could have been, I speculated [1] that it could be that the technology at the time was not up to handling the general case and so they regulat…

Most of us probably do the same thing when we read a HN comment about something specific: "This rando seems to know what they're talking about. I'll assume it as fact until I encounter otherwise." Not doing this might actually cause bigger problems... Getting first-hand experience or even reputable knowledge about something is extremely expensive compared to gut-checking random info you come across. So the "cheap kno…

I wish the source citing was more explicit. It would be great if the AI summary said something like, “almost no info about xyz can be found online but one GitHub comment says abc” (link)

Instead it often frames the answer as authoritative

Re: Two things LLM coding agents are still bad at

#348

Post by paulhebert »

schmichael wrote: post #343
kypro wrote: post #341

Earlier quoted context omitted.

But no one uses LLMs like this. This is the type of simple fact you could just Google and check yourself. LLMs are useful for providing answers to more complex questions where some reasoning or integration of information is needed. In these cases I mostly agree with the parent commenter. LLMs often come up with plausibly correct answers, then when you ask to cite sources they seem to just provide articles vaguely rel…

I do, and so does Google. When I googled "When was John Howard elected?" the correct answer came back faster in the AI Overview than I could find the answer in the results. The source the AI Overview links even provides confirmation of the correct answer.

Yeah but before AI overviews Google would have shown the first search result with a text snippet directly quoted from the page with the answer highlighted.

Thats just as fast (or faster) than the AI overview

Re: Two things LLM coding agents are still bad at

#349

Post by paulhebert »

ngruhn wrote: post #312
AIorNot wrote: post #282

Earlier quoted context omitted.

I think we need better code review tools in the age of LLMs - not just sticking another LLM to do a code review on top of the PR Needs to clearly handle the large diffs they produce - anyone have any ideas

I was about to write my own tool for this but then I discovered: git diff --color-moved=dimmed-zebra That shows a lot of code that was properly moved/copied in gray (even if it's an insertion). So gray stuff exactly matches something that was there before. Can also be enabled by default in the git config.

I would love if GitHub implemented this in their UI! There’s and issue: https://github.com/orgs/community/discussions/9632

Re: Two things LLM coding agents are still bad at

#350

Post by schmichael »

paulhebert wrote: post #348
schmichael wrote: post #343

Earlier quoted context omitted.

I do, and so does Google. When I googled "When was John Howard elected?" the correct answer came back faster in the AI Overview than I could find the answer in the results. The source the AI Overview links even provides confirmation of the correct answer.

Yeah but before AI overviews Google would have shown the first search result with a text snippet directly quoted from the page with the answer highlighted. Thats just as fast (or faster) than the AI overview

The snippet included in the search result does not include or highlight the relevant fact. I feel like you’re not willing to take simple actions to confirm your assertions.
Post reply on HN