Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two things LLM coding agents are still bad at

kix.dev

Post reply on HN
Print view View original

321–330 of 382 posts

Re: Two things LLM coding agents are still bad at

#321

Post by laterium »

grey-area wrote: post #313
laterium wrote: post #309

Earlier quoted context omitted.

They are lying, because their salary depends on them lying about it. Why does it even matter what they're saying? Why don't we listen to scientists, researchers, practicioners and the real users of the technology and stop repeating what the CEOs are saying? The things they're saying are technically correct, the best kind of correct. The models beat human PhDs on certain benchmarks of knowledge and reasoning. They may…

So questioning the utility of LLMs for knowledge work is now akin to a conspiracy theory?

Not what I said at all. Question it all what you want. But disproving outrageous CEO claims doesn't get you there. Whether LLMs are AGI/ASI that will replace everyone is seperate from whether they are useful today as tools. Attacking the first claim doesn't mean much for the second claim, which is the more interesting one.

Re: Two things LLM coding agents are still bad at

#322

Post by sarchertech »

NitpickLawyer wrote: post #305
sarchertech wrote: post #301

Earlier quoted context omitted.

>When's the last time you used one of these SotA coding agents? This morning :) >"so far outside of any capabilities" Anthropic was just bragging last week about being able to code without intervention for 30 hours before completely losing focus. They hailed it as a new bench mark. It completed a project that was 11k lines of code. The max unsupervised run that GPT-5-Codex has been able to pull off is 7 hours. That's…

I think you are confusing 2 things here. What the labs mean when they announce x hours sessions is on "one session" (i.e. the agent manages its own context via trimming and memory files, etc). What the project I linked did was "run in a bash loop", that basically resets the context every time the agent "finishes". That would mean that every few hours the agent starts fresh, does the inspect repo thing, does the plan…

There’s absolutely no difference from what he says he did and what Claude code can do behind the scenes.

If Anthropic thought they could produce anything remotely useful by wiping the context and reprompting every few hours, they would be doing it. And they’d be saying “look at this we implemented hard context reset and we can now run our agent for 30 days and produce an entire language implementation!”

In 3 months or 300 years of operating like this a current agent being freshly reprompted every few hours would never produce anything that even remotely looked like a language implementation.

As soon as its context was poisoned with slightly off topic todo comments it would spin out into writing a game of life implementation or whatever. You’d have millions of lines of nonsense code with nothing useful after 3 months of that.

The only way I see anything like this doing anything approaching “useful” is if the outer loop wipes the repo on every reset as well, and collects the results somewhere the agent can’t access. Then you essentially have 100 chances to one shot the thing.

But at that point you just have a needlessly expensive and slow agent.

Re: Two things LLM coding agents are still bad at

#323

Post by wvenable »

> LLMs are terrible at asking questions.

I was dealing with a particularly tricky problem in a technology I'm not super familiar with and GPT-5 eventually asked me to put in some debug code to analyze the state of the system as it ran. Once I provided it with the feedback it wanted, and a bit of back and forth, we were able to figure out what the issue was.

Re: Two things LLM coding agents are still bad at

#324

Post by erichocean »

ngruhn wrote: post #312
AIorNot wrote: post #282

Earlier quoted context omitted.

I think we need better code review tools in the age of LLMs - not just sticking another LLM to do a code review on top of the PR Needs to clearly handle the large diffs they produce - anyone have any ideas

I was about to write my own tool for this but then I discovered: git diff --color-moved=dimmed-zebra That shows a lot of code that was properly moved/copied in gray (even if it's an insertion). So gray stuff exactly matches something that was there before. Can also be enabled by default in the git config.

I used autochrome[0] for Clojure code to do this. (I also made some improvements to show added/removed comments, top-level form moves, and within-string/within-comment edits the way GitHub does.)

At first I didn't like the color scheme and replaced it with something prettier, but then I discovered it's actually nice to have it kinda ugly, makes it easier to detect the diffs.

[0] https://fazzone.github.io/autochrome.html

Re: Two things LLM coding agents are still bad at

#325

Post by neuronic »

tzs wrote: post #272

Just the other day I hit something that I hadn't realized could happen. It was not code related in my case, but could happen with code or code-related things (and did to a coworker). In a discussion here on HN about why a regulation passed 15 years ago was not as general as it could have been, I speculated [1] that it could be that the technology at the time was not up to handling the general case and so they regulat…

I even curated a list of 6-8 sources in NotebookLM recently, asked a very straight-forward question (which credential formats does OID4VP allow). The sources were IETF and OpenID specs + some additional articles on it.

I wanted to use NotebookLM as a tool to ask back and forth when I was trying to understand stuff. It got the answer 90% right but also added a random format, sounding highly confident as if I asked the spec authors themselves.

It was easy to check the specs when I became suspicious and now my trust, even in "grounded" LLMs, is completely eroded when it comes to knowledge and facts.

Re: Two things LLM coding agents are still bad at

#326

Post by cpfohl »

ravila4 wrote: post #187
cpfohl wrote: post #172

Earlier quoted context omitted.

My custom prompt instructs GPT to output changes to code as a diff/git-patch. I don’t use agents because it makes it hard to see what’s happening and I don’t trust them yet.

I’ve tried this approach when working in chat interfaces (as opposed to IDEs), but I often find it tricky to review diffs without the full context of the codebase. That said, your comment made me realize I could be using “git apply”more effectively to review LLM-generated changes directly in my repo. It’s actually a neat workflow!

Yep!! It’s fantastic

Re: Two things LLM coding agents are still bad at

#327

Post by baq »

grey-area wrote: post #59
ivape wrote: post #53

Earlier quoted context omitted.

You’re just not using LLMs enough. You can never trust the LLM to generate a url, and this was known over two years ago. It takes one token hallucination to fuck up a url. It’s very good at a fuzzy great answer, not a precise one. You have to really use this thing all the time and pick up on stuff like that.

Or just not bother. It sounds pretty useless if it flunks on basic tasks like this. Perhaps you’ve been sold a lie?

Read about the jagged frontier. IanCal is right: this is a perfect example of using the tool wrong; you’ve focused on a very narrow use case which is surprisingly hard for the matmuls to not mess up and extrapolate, but extrapolation is incorrect here because the capability frontier is fractal and not continuous.

Re: Two things LLM coding agents are still bad at

#328

Post by BeetleB »

linsomniac wrote: post #198

>Sure, you can overengineer your prompt to try get them to ask more questions That's not overengineering, that's engineering. "Ask clarifying questions before you start working", in my experience, has led to some fantastic questions, and is a useful tool even if you were to not have the AI tooling write any code. As a good programmer, you should know when you are handing the tool a complete spec to build the code and…

Yeah, this made me stop reading. I often tell it to ask me any questions if unclear (and sometimes my prompt is just "Hey, this is my idea. Ask me questions to flesh it out").

It always asks me questions, and I've always benefited from it. It will subtly point out things I hadn't thought about, etc.

Re: Two things LLM coding agents are still bad at

#329

Post by baq »

parineum wrote: post #252
buildbot wrote: post #227

Earlier quoted context omitted.

Would you hire a PhD to copy URLs by hand? Would them having PhD make it less likely they’d make a mistake than an high school student doing the same?

A high school student would use copy/paste and the urls would be perfect duplicates..

LLMs aren’t high school students, they’re blobs of numbers which happen to speak English if you poke them right. Use the tool when it’s good at what it does.

Re: Two things LLM coding agents are still bad at

#330

Post by AlwaysRock »

tzs wrote: post #272

Just the other day I hit something that I hadn't realized could happen. It was not code related in my case, but could happen with code or code-related things (and did to a coworker). In a discussion here on HN about why a regulation passed 15 years ago was not as general as it could have been, I speculated [1] that it could be that the technology at the time was not up to handling the general case and so they regulat…

Asking for a source from llms is so eye opening. I am yet to have them link a source that actually supports what they said.
Post reply on HN