Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

451–460 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#451

Post by tovej »

satvikpendem wrote: post #308
tovej wrote: post #299

Earlier quoted context omitted.

Because they don't. The chain-of-reasoning feature is really just a way to get the LLM to prompt more. The fact that it generates these "thinking" steps does not mean it is using them for reasoning. It's most useful effect is making it seem to a human that there is a reasoning process.

How would you determine humans have reasoning then, in a way that LLMs do not?

Easy, humans can synthesize new facts using logic and context. And the conclusions can be checked against the real world to check that the reasoning was correct.

Or another way: reasoning is a socially constructed concept, developed by humans. Humans therefore have defined reasoning, and must therefore know how to reason.

Or a third way: I experience reasoning, you experience reasoning. I am currently reasoning. You are currently reasoning. I am human, as are you. Therefore humans reason.

Re: LLMs work best when the user defines their acceptance criteria first

#452

Post by tovej »

andai wrote: post #326
satvikpendem wrote: post #308

Earlier quoted context omitted.

How would you determine humans have reasoning then, in a way that LLMs do not?

Or — here's a fun one — subjective experience.

This one is even easier. LLMs record objective data about n-gram distribution, there is no room for any "subjective state" in their working set.

Or another way: an LLM will respond the same way if you wait for one second or a decade between prompts. The only way it is interacted with is through a stream of tokens. There is exactly one stream at all times, and each time the stream is input again, it is barely different from the previous input. The LLM does not behave differently depending on the contents of the stream. It may produce the exact same token for two different streams. It may also encounter the same stream twice, and will act the same in both cases. If it were a "subject" "experiencing" say, a discussion, it would use its pasts "experience". But it does not.

Re: LLMs work best when the user defines their acceptance criteria first

#453

Post by tovej »

seanmcdirmid wrote: post #416
tovej wrote: post #411

Earlier quoted context omitted.

You completely misunderstood what I wrote. It was a response to you saying: "Im not going around trying to prove my code correct when I write it manually." How did you manage to forget what you wrote previously? Also, in this post you are now suddenly taking the exact opposite position, contradicting your previous point.

I did not contradict my previous point. But now I’m confused in how you think we use LLMs to write code. You made it sound like we just get it todump out code without any process in between.

You most definitely did contradict yourself. First you said you don't prove anything about the code you write, then you said you do. But that's fine. We can agree to disagree.

And I have not made any statements about how you use LLMs, only about how the LLMs produce code. All statements about how you use LLMs have been made by you, not me. I haven't discussed it since it is not related to the arguments, which are: 1) whether LLMs are goal-oriented and 2) whether humans and LLMs both merely maximize plausibility when writing/generating code.

Both claims that you made. Note, however, that if you are correct in your own points, then you should indeed be able to "just dump out code without any process in between". So if anyone is claiming this, it's you.

Re: LLMs work best when the user defines their acceptance criteria first

#454

Post by nickcoffee »

The acceptance criteria point translates directly outside of coding too. Using Claude Code for sales and operational workflows, having acceptable criteria upfront (along with some manual checks along the way depending on the task) definitely helps the output.

Re: LLMs work best when the user defines their acceptance criteria first

#455

Post by KatanaLarp »

serious_angel wrote: post #29

Holy gracious sakes... Of course... Thank you... thank you... dear katanaquant, from the depths... of my heart... There's still belief in accountability... in fun... in value... in effort... in purpose... in human... in art... Related: - http://archive.today/2026.03.07-020941/https://lr0.org/blog/... > (I'm not consulting an LLM...) - https://web.archive.org/web/20241021113145/https://slopwatch... >

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#456

Post by dimaggiosghost »

marginalia_nu wrote: post #378
Implicated wrote: post #366

Earlier quoted context omitted.

I've had very consistent success with plan mode, but when I haven't I've noticed many times it's been working with code/features/things that aren't well defined. ie: not using a well defined design pattern, maybe some variability in the application on how something could be done - these are the things I notice it really trips up on. Well defined interfaces, or even specifically telling it to identify and apply design…

Ok so here are the actual course corrections I had to make to push through a replacement implementation of a btree. Note that almost all of the problems aren't with the implementation, it basically one shot that. Almost all the issues are with integrating the change with the wider system. "The btree library is buggy, and inefficient (using mmap, a poor design idea). Can you extract it to an interface, and then implem…

One thing I'm finding early success with is to define how the system can know if this statement is being met. Frequently I will include in the prompt e.g. "research what makes good high quality engineering practices and derive how to tell if those practices are being followed".

Directly telling it my team's values would be better, if we have it developed (like the style guide you mentioned) ... but that's a lot of work, the reasons that hasn't happened before are just as true now, and honestly - there's a lot of overlap with the generic research result.

> Are these changes good, high quality, good engineering practices, in line with known best practices and the style guide

Re: LLMs work best when the user defines their acceptance criteria first

#457

Post by 0x008 »

stingraycharles wrote: post #46
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

> If you ask to unify the duplication, it'll say "No problem, here's a brand new metamock abstract adapter framework that has a superset of all feature sets, plus two new metamock drivers for the older and the newer code! Let me know if you want me to write tests for the new adapters." Nevermind the fact that it only migrated 3 out of 5 duplicated sections, and hasn’t deleted any now-dead code.

The problem is that you are looking at the code. /s

Re: LLMs work best when the user defines their acceptance criteria first

#458

Post by oofbey »

2god3 wrote: post #368
oofbey wrote: post #342

Earlier quoted context omitted.

If it wasn’t clear, I don’t have any desire to convince anybody of anything. You don’t believe the future is here yet? Good luck holding on to that position. Not my problem. I was taking time to try to help somebody who sounded genuinely curious and seeking help. That I’m happy to do.

You’re writing novels when if you had something compelling to show it’d be simple and easy. If you can’t make it simple and easy… then you haven’t understood it at all. All geniuses refer to this as the standard by which one understands something. Whether it’s Steve Jobs or Einstein. So don’t get mad. Show us all how simple and easy it is. If you can’t.. then accept you’re full of it and don’t quite get it as well as…

You’re not going to create any future if you insult people trying to offer friendly advice, or think of the talent you rely on to create your vision as “bozo programmers”. I’d wish you good luck, but you have convinced me you don’t deserve it.

Re: LLMs work best when the user defines their acceptance criteria first

#459

Post by hkonte »

Implicated wrote: post #88
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#460

Post by Shyaamal11 »

One thing I’ve noticed while working with data/AI workflows is that the “acceptance criteria first” idea applies even more strongly once you move beyond code generation into data pipelines and analytics.

LLMs can generate queries, transformations, or even Spark jobs that look reasonable but if the underlying data contracts, schema expectations, or evaluation criteria aren’t defined, you end up with something that looks correct but is semantically wrong.

In practice, the teams that get the most value from AI-assisted development tend to have: clearly defined datasets reproducible data pipelines well-defined outputs / metrics Once those pieces are in place, AI becomes much more useful because it’s operating inside a structured system instead of guessing context. That’s also why there’s been a lot of interest lately in lakehouse-style platforms that combine data engineering, analytics, and AI workflows in one place (e.g. platforms like IOMETE).

When the data layer is structured and reproducible, AI tooling becomes far more reliable. Curious if others here have seen the same pattern when using LLMs for data engineering or analytics work.

Post reply on HN