Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

151–160 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#151

Post by MadnessASAP »

leke wrote: post #138
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

i wonder if the solution is to just ask it to refactor its code once it's working.

You can, and it might make things a bit better. The only real way I've found so far is to start going through file by file, picking it apart.

I wouldn't be surprised if over half my prompts start with "Why ...?", usually followed by "Nope, ... instead”

Maybe the occasional "Fuck that you idiot, throw the whole thing out"

Re: LLMs work best when the user defines their acceptance criteria first

#152

Post by evolve-maz »

Jweb_Guru wrote: post #121
Foobar8568 wrote: post #114

Earlier quoted context omitted.

That's the reality nobody really wants to say.

It's not reality. I'm really not a fan of the way that people excuse the really terrible code LLMs write by claiming that people write code just as bad. Even if that were true, it is not true that when you ask those people to do otherwise they simply pretend to have done it and forget you asked later.

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#153

Post by iLoveOncall »

Implicated wrote: post #129
Foobar8568 wrote: post #125

Earlier quoted context omitted.

At least my comment hasn't been reviewed or written by a LLM. And in my French brain, code or codebase is countable and not uncountable.

I got curious and had to fire up the ol LLM to find out what the story is about the words that aren't pluralized - TIL about countable and uncountable nouns. I wonder if the guy giving you trouble about your English speaks French.

I'm native French and nobody would consider code countable. "codes" makes no sense. We'd talk about "lines of code" as a countable in French just like in English.

Re: LLMs work best when the user defines their acceptance criteria first

#154

Post by iLoveOncall »

unlikelytomato wrote: post #70
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

This is why I'm confused when people say it isn't ready to replace most of the programmer workforce.

Really? Because this perfectly explains why it will never replace them: it needs an exact language listing everything required to function as you expect it.

You need code to get it to generate proper code.

Re: LLMs work best when the user defines their acceptance criteria first

#155

Post by ttoinou »

Jweb_Guru wrote: post #121
Foobar8568 wrote: post #114

Earlier quoted context omitted.

That's the reality nobody really wants to say.

It's not reality. I'm really not a fan of the way that people excuse the really terrible code LLMs write by claiming that people write code just as bad. Even if that were true, it is not true that when you ask those people to do otherwise they simply pretend to have done it and forget you asked later.

No but they will despise you for bringing the problem up

Re: LLMs work best when the user defines their acceptance criteria first

#156

Post by simianwords »

ozozozd wrote: post #92
D-Machine wrote: post #90

This article is great. And the blog-article headline is interesting, but wrong. LLM's don't in general write plausible code (as a rule) either. They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR. This isn't that hard to remember, and is a correct enough simplification of what generative LLMs actually do, without resorting…

Exactly. It’s also easy to find yourself in the out-of-distribution territory. Just ask for some tree-sitter queries and watch Gemini 3, Opus 4.5 and GLM 5 hallucinate new directives.

Any example of how I can get it to hallucinate?

Re: LLMs work best when the user defines their acceptance criteria first

#157

Post by grey-area »

This is a fascinating look into code generated by an LLM that is correct in one sense (passes tests) but doesn't meet requirements (painfully slow). Doesn't use is_ipk to identify primary keys, uses fsync on every statement. The problem with larger projects like this even if you are competent is that there are just too many lines of code to read it properly and understand it all. Bravo to the author for taking the time to read this project, most people never will (clearly including the author of it).

I find LLMs at present work best as autocomplete -

The chunks of code are small and can be carefully reviewed at the point of writing

Claude normally gets it right (though sometimes horribly wrong) - this is easier to catch in autocomplete

That way they mostly work as designed and the burden on humans is completely manageable, plus you end up with a good understanding of the code generated. They make mistakes I'd say 30% of the time or so when autocompleting, which is significant (mistakes not necessarily being bugs but ugly code, slow code, duplicate code or incorrect code.

Having the AI produce the majority of the code (in chats or with agents) takes lots of time to plan and babysit, and is harder to review, maintain and diagnose; it doesn't seem like much of a performance boost, unless you're producing code that is already in the training data and just want to ignore the licensing of the original code.

Re: LLMs work best when the user defines their acceptance criteria first

#158

Post by simianwords »

jqpabc123 wrote: post #48

LLMs have no idea what "correct" means. Anything they happen to get "correct" is the result of probability applied to their large training database. Being wrong will always be not only possible but also likely any time you ask for something that is not well represented in it's training data. The user has no way to know if this is the case so they are basically flying blind and hoping for the best. Relying on an LLM f…

This is easily proven incorrect. Just go to ChatGPT and say something incorrect and ask it to verify. Why do people still believe this type of thing?

Re: LLMs work best when the user defines their acceptance criteria first

#159

Post by zabzonk »

riffraff wrote: post #126
Implicated wrote: post #88

Earlier quoted context omitted.

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

> Whew. Ok. You don't tell it the code is slow. Do you tell your coworker "Hey, your code is slow" and expect great results? Yes? Why don't you? They are capable people that just didn't notice something, id I notice some telemetry and tell them "hey this is slow" they are expected to understand the reason(s).

Well, I would say something like "We seem to be having some performance issues the business has noticed in the XYZ stuff. Shall we sit down together and see if we can work out if we can improve things?"

Re: LLMs work best when the user defines their acceptance criteria first

#160

Post by mentalgear »

> I write this as a practitioner, not as a critic. After more than 10 years of professional dev work, I’ve spent the past 6 months integrating LLMs into my daily workflow across multiple projects. LLMs have made it possible for anyone with curiosity and ingenuity to bring their ideas to life quickly, and I really like that! But the number of screenshots of silently wrong output, confidently broken logic, and correct-looking code that fails under scrutiny I have amassed on my disk shows that things are not always as they seem.

Same experience, but the hype bros do only need a shiny screengrab to proclaim the age of "gatekeeping" SWE is over to get their click fix from the unknowingly masses.

Post reply on HN