Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

431–440 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#431

Post by jaredklewis »

unlikelytomato wrote: post #70
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

This is why I'm confused when people say it isn't ready to replace most of the programmer workforce.

This comment is why I read HN

Re: LLMs work best when the user defines their acceptance criteria first

#432

Post by jamesblonde »

The reference in the text to Anthropic’s “Towards Understanding Sycophancy in Language Models” is related to RLHF (reinforcement learning with human feedback).

Claude code uses primarily different "pathways" in Anthropic LLMs that were not post-trained with RLHF, but rather with RLVF (reinforcement learning with verifiable rewards).

So, his point about code being produced to please the user isn't valid from where I am sitting.

Re: LLMs work best when the user defines their acceptance criteria first

#433

Post by theshrike79 »

roarcher wrote: post #430
theshrike79 wrote: post #423

Earlier quoted context omitted.

If code smells like LLM, then you walk to said coworker and ask them to explain it for you. Play dumb if necessary. Or you use YOUR LLM to review the PR :D ...and wtf, you get "credited" story points for finishing tasks? That sounds completely insane.

> you get "credited" story points for finishing tasks? That sounds completely insane. Developers' names are attached to stories, and stories have points on them. Why is that insane, and how does your company track who did what? I propose that the name on the story should be that of the reviewer since they did the work.

We don't really track individual features to people in a way I could call "crediting" - as in nobody really checks afterwards who did how many story points in a sprint.

As long as the team as a whole gets stuff done, everything is good.

Re: LLMs work best when the user defines their acceptance criteria first

#434

Post by treetalker »

otterley wrote: post #383
treetalker wrote: post #244

This is my experience with how LLMs "draft" legal arguments: at first glance, it's plausible — but may be, and often is, invalid, unsound, and/or ill-advised. The catch is that many judges lack the time, energy, or willingness to not only read the documents in detail, but also roll up their sleeves and dig into the arguments and cited authorities. (Some lack the skills, but those are extreme cases.) So the plausible…

As an attorney, I’m interested in this theory. Do you have any examples that illustrate the phenomenon you describe?

Since it's too late to edit my other reply — here is a description of a recent case involving several of the categories I mentioned: https://reason.com/volokh/2026/03/06/california-appeals-cour...

And you will find many more by reading (or subscribing to the RSS feed of) the Volokh Conspiracy blog's "AI in Court" tag: https://reason.com/category/law/ai-in-court/

Re: LLMs work best when the user defines their acceptance criteria first

#435

Post by KatanaLarp »

grey-area wrote: post #157

This is a fascinating look into code generated by an LLM that is correct in one sense (passes tests) but doesn't meet requirements (painfully slow). Doesn't use is_ipk to identify primary keys, uses fsync on every statement. The problem with larger projects like this even if you are competent is that there are just too many lines of code to read it properly and understand it all. Bravo to the author for taking the ti…

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#436

Post by KatanaLarp »

jamesblonde wrote: post #432

The reference in the text to Anthropic’s “Towards Understanding Sycophancy in Language Models” is related to RLHF (reinforcement learning with human feedback). Claude code uses primarily different "pathways" in Anthropic LLMs that were not post-trained with RLHF, but rather with RLVF (reinforcement learning with verifiable rewards). So, his point about code being produced to please the user isn't valid from where I a…

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#437

Post by KatanaLarp »

cadamsdotcom wrote: post #413

This is a bit unfair - to generate a bunch of code but not give the model data/tools and direct it to optimize it; then compare it to the optimized work of thousands over decades. Feels like an extremely high effort hit piece, even though I know it’s not.

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#438

Post by KatanaLarp »

einrealist wrote: post #147

> SQLite is not primarily fast because it is written in C. Well.. that too, but it is fast because 26 years of profiling have identified which tradeoffs matter. Someone (with deep pockets to bear the token costs) should let Claude run for 26 months to have it optimize its Rust code base iteratively towards equal benchmarks. Would be an interesting experiment. The article points out the general issue when discussing L…

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#440

Post by khy34 »

godelski wrote: post #406
jghn wrote: post #376

Earlier quoted context omitted.

Absolutely. Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications. These are *not* the people I'm talking about now having superpowers. I am picturing people who blend high level engineering and product skills, ideally with business sense.

> Merely being a member of the business class does not magically mean one has the ability to specify business requirements much less product specifications Is this not why COBOL failed? Common Business-Oriented Language sure does look much more like natural language than a lot of other code, but it could never solve the abstraction needed to do the complex things. I don't think LLMs will ever get rid of coders. Busin…

My intuition tells me that llm’s combined with SWE’s with really amazing fundamentals will kill the code monkeys.

And frankly? That’s the best outcome. Code monkeys (in my view that’s an individual who writes out code just to complete a jira ticket) are a liability. Not only that but each additional person you have in an org means more noise creation.

If this forces the code monkeys to level up to compete… again a good thing.

The code base should not be elongated nor complicated. I’m not even a SWE by trade, rather a CEO, and this is my preferred outcome.

Post reply on HN