Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two things LLM coding agents are still bad at

kix.dev

Post reply on HN
Print view View original

101–110 of 382 posts

Re: Two things LLM coding agents are still bad at

#101

Post by jeswin »

iLoveOncall wrote: post #84
jeswin wrote: post #68

Earlier quoted context omitted.

> I wanted it to refactor a parser in a small project This expression tree parser (typescript to sql query builder - https://tinqerjs.org/ ) has zero lines of hand-written code. It was made with Codex + Claude over two weeks (part-time on the side). Having worked on ORMs previously, it would have taken me 4x-10x the time to get to the same state (which also has 100s of tests, with some repetitions). That's a massive…

Hum yeah, it shows. Just the fact that the API looks completely different for Postgre and SQLite tells us everything we need to know about the quality of the project here.

> Just the fact that the API looks completely different for Postgre and SQLite tells us everything we need to know about the quality of the project here.

How does the API look completely different for pg and sqlite? Can you share an example?

It's an implementation of LINQ's IQueryable. With some bells missing in DotNet's Queryable, like Window functions (RANK queries etc) which I find quite useful.

Add: What you've mentioned is largely incorrect. But in any case, it is a query builder. Meaning, an ORM like database abstraction is not the goal. This allows us to support pg's extensions, which aren't applicable to other database.

Re: Two things LLM coding agents are still bad at

#102

Post by regularfry »

freetonik wrote: post #34

I see a pattern in these discussions all the time: some people say how very, very good LLMs are, and others say how LLMs fail miserably; almost always the first group presents examples of simple CRUD apps, frontend "represent data using some JS-framework" kind of tasks, while the second group presents examples of non-trivial refactoring, stuff like parsers (in this thread), algorithms that can't be found in leetcode,…

The function of technological progress, looked at through one lens, is to commoditise what was previously bespoke. LLMs have expanded the set of repeatable things. What we're seeing is people on the one hand saying "there's huge value in reducing the cost of producing rote assets", and on the other "there is no value in trying to apply these tools to tasks that aren't repeatable".

Both are right.

Re: Two things LLM coding agents are still bad at

#103

Post by jsjshxnsbs »

ivape wrote: post #90
hansmayer wrote: post #89

Earlier quoted context omitted.

Yeah so, the reason people use various tools and machines in the first place is to simplify the work or everydays tasks by : 1) Making the tasks execute faster 2) Getting more reliable outputs then doing this by yourself 3) Making it repeatable . The LLMs obviously dont check any of these boxes so why don´t we stop pretending that we as users are stupid and don´t know how to use them and start taking them for what th…

[flagged]

I'll take the L when llms can actually do my job to the level I expect. Llms can do some of my work but they are tiring they make mistakes and they absolutely get confused by a sufficiently complex and large codebase.

Quite frankly, not being able to discuss the pros and the cons of a technology with other engineers absolutely hinders innovation. A lot of discoveries come out of mistakes.

Stop being so small minded.

Re: Two things LLM coding agents are still bad at

#104

Post by sidgtm »

seunosewa wrote: post #88
sidgtm wrote: post #36

As a UX designer I see they lack the ability of being opinionated about a design piece and go with the standard mental model. I got fed up with this and made a simple java script code to run a simple canvas on the localhost to pass on more subjective feedback using highlights and notes feature. I tried using playwright first but a. its token heavy b. it's still for finding what's working or breaking instead of thinki…

Whatdo the notes look like?

specific inputs e.g. move, color change, or giving specific inputs for interaction piece.

Re: Two things LLM coding agents are still bad at

#105

Post by raffael_de »

coldtea wrote: post #76
rossant wrote: post #37

Recently, I asked Codex CLI to refactor some HTML files. It didn't literally copy and pasted snippets here and there as I would have done myself, it rewrote them from memory, removing comments in the process. There was a section with 40 successive links with complex URLs. A few days later, just before deployment to production, I wanted to double check all 40 links. First one worked. Second one worked. Third one worke…

> A few days later, just before deployment to production, I wanted to double check all 40 links. This was allowed to go to master without "git diff" after Codex was done?

This and why are the URLs hardcoded to begin with? And given the chaotic rewrite by Codex it would probably be more work to untangle the diff than just do it yourself right away.

Re: Two things LLM coding agents are still bad at

#106

Post by Leynos »

cheema33 wrote: post #71

From the article: > I contest the idea that LLMs are replacing human devs... AI is not able to replace good devs. I am assuming that nobody sane is claiming such a thing today. But, it can probably replace bad and mediocre devs. Even today. In my org we had 3 devs who went through a 6-month code boot camp and got hired a few years ago when it was very difficult to find good devs. They struggled. I would give them eas…

What do you think was the reason that the bootcamp grads struggling to get better at what they do?

Re: Two things LLM coding agents are still bad at

#107

Post by cimi_ »

worldsayshi wrote: post #69
rossant wrote: post #37

Recently, I asked Codex CLI to refactor some HTML files. It didn't literally copy and pasted snippets here and there as I would have done myself, it rewrote them from memory, removing comments in the process. There was a section with 40 successive links with complex URLs. A few days later, just before deployment to production, I wanted to double check all 40 links. First one worked. Second one worked. Third one worke…

This is of course bad but: humans also makes (different) mistakes all the time. We could account for the risk of mistakes being introduced and make more tools that validate things for us. In a way LLM:s encourage us to do this by adding other vectors of chaos into our work. Like, why not have tools built into our environment that checks that links are not broken? With the right architecture we could have validations…

Your point to not rely on good intentions and have systems in place to ensure quality is good - but your comparison to humans didn't go well with me.

Very few humans fill in their task with made up crap then lie about it - I haven't met any in person. And if I did, I wouldn't want to work with them, even if they work 24/7.

Obligatory disclaimer for future employers: I believe in AI, I use it, yada yada. The reason I'm commenting here is I don't believe we should normalise this standard of quality for production work.

Re: Two things LLM coding agents are still bad at

#108

Post by Leynos »

tjansen wrote: post #18

Agreed with the points in that article, but IMHO the no 1 issue is that agents only see a fraction of the code repository. They don't know whether there is a helper function they could use, so they re-implement it. When contributing to UIs, they can't check the whole UI to identify common design patterns, so they re-invent it. The most important task for the human using the agent is to provide the right context. "Loo…

I wonder if a large context model could be employed here via tool call. One of the great things Gemini chat can do is ingest a whole GitHub repo.

Perhaps "before implementing a new utility or helper function, ask the not-invented-here tool if it's been done already in the codebase"

Of course, now I have to check if someone has done this already.

Re: Two things LLM coding agents are still bad at

#109

Post by pprotas »

iLoveOncall wrote: post #84
jeswin wrote: post #68

Earlier quoted context omitted.

> I wanted it to refactor a parser in a small project This expression tree parser (typescript to sql query builder - https://tinqerjs.org/ ) has zero lines of hand-written code. It was made with Codex + Claude over two weeks (part-time on the side). Having worked on ORMs previously, it would have taken me 4x-10x the time to get to the same state (which also has 100s of tests, with some repetitions). That's a massive…

Hum yeah, it shows. Just the fact that the API looks completely different for Postgre and SQLite tells us everything we need to know about the quality of the project here.

I guess the interesting question is whether @jeswin could have created this project at all if AI tools were not involved. And if yes, would the quality even be better?

Re: Two things LLM coding agents are still bad at

#110

Post by rdsubhas »

tjansen wrote: post #18

Agreed with the points in that article, but IMHO the no 1 issue is that agents only see a fraction of the code repository. They don't know whether there is a helper function they could use, so they re-implement it. When contributing to UIs, they can't check the whole UI to identify common design patterns, so they re-invent it. The most important task for the human using the agent is to provide the right context. "Loo…

To be fair, this is a daily life story for any senior engineer working with other engineers.
Post reply on HN