Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two things LLM coding agents are still bad at

kix.dev

Post reply on HN
Print view View original

331–340 of 382 posts

Re: Two things LLM coding agents are still bad at

#331

Post by braebo »

gengstrand wrote: post #276

The conversation here seems to be more focused on coding from scratch. What I have noticed when I was looking at this last year was that LLMs were bad at enhancing already existing code (e.g. unit tests) that used annotation (a.k.a. decorators) for dependency injection. Has anyone here attempted that with the more recent models? If so, then what were your findings?

My experience is the opposite. The latest Claude seems to excel in my personal medium-sized (20-50k loc) codebases with strong existing patterns and a robust structure from which it can extrapolate new features or documentation. Claude Code is getting much better at navigating code paths across many large files in order to provide nuanced and context-aware suggestions or bug fixes.

When left to its own devices on tasks with little existing reference material to draw from, however, the quality and consistency suffers significantly and brittle, convoluted structures begin to emerge.

This is just my limited experience though, and I almost never attempt to, for example, vibe-code an entire greenfield mvp.

Re: Two things LLM coding agents are still bad at

#333

Post by nberkman »

Inspired by the copy-paste point in this post, I added agent buffer tools to clippy, a macOS utility I maintain which includes an MCP server that interacts with the system clipboard. In this case it was more appropriate to use a private buffer instead. With the tools I just added, the server reads file bytes directly - your agent never generates the copied content as tokens. Three operations:

buffer_copy: Copy specific line ranges from files to agent's private buffer

buffer_paste: Insert/append/replace those exact bytes in target files

buffer_list: See what's currently buffered

So the agent can say "copying lines 50-75 from auth.py" and the MCP server handles the actual file I/O. No token generation, no hallucination, byte-for-byte accurate. Doesn't touch your system clipboard either.

The MCP server already included tools to copy AI-generated content to your system clipboard - useful for "write a Python script and copy it" workflows.

(Clippy's main / original purpose is improving on macOS pbcopy - it copies file references instead of just file contents, so you can paste actual files into Slack/email/etc from the terminal.)

If you're on macOS and use Claude or other MCP-compatible agents: https://github.com/neilberkman/clippy

brew install neilberkman/clippy/clippy

Re: Two things LLM coding agents are still bad at

#334

Post by willsmith72 »

AlwaysRock wrote: post #330
tzs wrote: post #272

Just the other day I hit something that I hadn't realized could happen. It was not code related in my case, but could happen with code or code-related things (and did to a coworker). In a discussion here on HN about why a regulation passed 15 years ago was not as general as it could have been, I speculated [1] that it could be that the technology at the time was not up to handling the general case and so they regulat…

Asking for a source from llms is so eye opening. I am yet to have them link a source that actually supports what they said.

> I am yet to have them link a source that actually supports what they said.

You're not trying very hard then. Here, my first try: https://claude.ai/share/ef7764d3-6c5c-4d1a-ba28-6d5218af16e0

Re: Two things LLM coding agents are still bad at

#336

Post by HarHarVeryFunny »

Romario77 wrote: post #222
dkarl wrote: post #208

Earlier quoted context omitted.

I've had similar experience both in coding and in non-coding research questions. An LLM will do the first N right and fake its work on the rest. It even happens when asking an LLM to reformat a document, or asking it to do extra research to validate information. For example, before a recent trip to another city, I asked Gemini to prepare a list of brewery taprooms with certain information, and I discovered it had inc…

LLMs are not good at "cycles" - when you have to go over a list and do the same action on each item. It's like it has ADHD and forgets or gets distracted in the middle. And the reason for that is that LLMs don't have memory and process the tokens, so as they keep going over the list the context becomes bigger with more irrelevant information and they can lose the reason they are doing what they are doing.

Right.

In a recent YouTube interview Karpathy claimed that LLMs have a lot more "working memory" than a human:

https://www.youtube.com/watch?v=hM_h0UA7upI&t=1306s

What I assume he's talking about is internal activations such as stored in KV cache that have same lifetime as tokens in the input, but this really isn't the same as "working memory" since these are tied to the input and don't change.

What it seems an LLM would need to do better at these sort of iterative/sequencing tasks would be a real working memory that had more arbitrary task-duration lifetime and could be updated (vs fixed KV cache), and would allow it to track progress or more generally maintain context (english usage - not LLM) over the course of a task.

I'm a bit surprised that this type of working memory hasn't been added to the transformer architecture. It seems it could be as simple as a fixed (non shifting) region of the context that the LLM could learn to read/write during training to assist on these types of task.

An alternative to having embeddings as working memory is to use an external file of text (cf a TODO list, or working notes) for this purpose which is apparently what Claude Code uses to maintain focus over long periods of time, and I recently saw mentioned that the Claude model itself has been trained to use read/write to this sort of text memory file.

Re: Two things LLM coding agents are still bad at

#338

Post by alfalfasprout »

AIorNot wrote: post #282
scottbez1 wrote: post #247

Earlier quoted context omitted.

The last point I think is most important: "very subtle and silently introduced mistakes" -- LLMs may be able to complete many tasks as well (or better) than humans, but that doesn't mean they complete them the same way, and that's critically important when considering failure modes. In particular, code review is one layer of the conventional swiss cheese model of preventing bugs, but code review becomes much less eff…

I think we need better code review tools in the age of LLMs - not just sticking another LLM to do a code review on top of the PR Needs to clearly handle the large diffs they produce - anyone have any ideas

Yep, this pattern of LLMs reviewing LLMs is terrifying to me. It's literally the inmates running the asylum.

Re: Two things LLM coding agents are still bad at

#339

Post by ACow_Adonis »

tzs wrote: post #272

Just the other day I hit something that I hadn't realized could happen. It was not code related in my case, but could happen with code or code-related things (and did to a coworker). In a discussion here on HN about why a regulation passed 15 years ago was not as general as it could have been, I speculated [1] that it could be that the technology at the time was not up to handling the general case and so they regulat…

I find it much more intuitive to think of LLMs as fuzzy-indexed frequency based searches combined with grammatically correct probabilistic word generators.

They have no concept of truth or validity, but the frequency of inputs into their training data provides a kind of psuedo check and natural approximation to truth as long as frequency and relationships in the training data also has some relationship to truth.

For a lot of textbook coding type stuff that actually holds: frameworks, shell commands, regexes, common queries and patterns. There's lots of it out there and generally the more common form is spreading some measure of validity.

My experience though is that on niche topics, sparse areas, topics that humans are likely to be emotionally or politically engaged with (and therefore not approximate truth), or things that are recent and therefore haven't had time to generate sufficient frequency, they can get thrown off. And of course it also has no concept of whether what it is finding or reporting is true or not.

This also explains why they have trouble with genuine new programming and not just reimplementing frameworks or common applications because they lack the frequency based or probabilistic grounding to truth and because the new combinations of libraries and code leads to place of relative sparsity in it's weights that leave them unable to function.

The literature/marketing has taken to calling this hallucination, but it's just as easy to think of it as errors produced by probabilistic generation and/or sparsity.

Re: Two things LLM coding agents are still bad at

#340

Post by purplerabbit »

tzs wrote: post #272

Just the other day I hit something that I hadn't realized could happen. It was not code related in my case, but could happen with code or code-related things (and did to a coworker). In a discussion here on HN about why a regulation passed 15 years ago was not as general as it could have been, I speculated [1] that it could be that the technology at the time was not up to handling the general case and so they regulat…

Most of us probably do the same thing when we read a HN comment about something specific: "This rando seems to know what they're talking about. I'll assume it as fact until I encounter otherwise."

Not doing this might actually cause bigger problems... Getting first-hand experience or even reputable knowledge about something is extremely expensive compared to gut-checking random info you come across. So the "cheap knowledge" may be worth it on balance.

Post reply on HN