Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

351–360 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#351

Post by _0ffh »

jghn wrote: post #329
_0ffh wrote: post #320

Earlier quoted context omitted.

Might be a big company thing then, but I'm not wholly convinced. There's a big gap between designing the outline of a big system and coding instructions that can be followed without having to make your own decisions. The question of how much of that gap is filled by the "design" vs "coding" levels is a spectrum.

I think I see what you're saying and if so we're talking past each other a bit and I agree with what you're saying as well. The point I was raising is by the time an IC developer sees something, there's already been a process of curation that happens that frames the possible solutions & constrains branch points. This is different from saying that an IC makes 0 implementation decisions. The C-suite has set a direction…

I'm on board with that framing of the process, and I see how my original formulation was too rough.

I was reacting to "We're moving to a world where the mechanical part of grinding the code is not worth much". I have the impression that in the past just mechanically grinding the code was less of a thing than it apparently is today. Guidance, sure, but not as much as seems to be common (often necessarily so) today. But I'm sure that varies with a lot of factors, not just the calendar year.

Re: LLMs work best when the user defines their acceptance criteria first

#352

Post by ramoz »

plandis wrote: post #350
msvana wrote: post #346

I think there is one problem with defining acceptance criteria first: sometimes you don't know ahead of time what those criteria are. You need to poke around first to figure out what's possible and what matters. And sometimes the criteria are subjective, abstract, and cannot be formally specified. Of course, this problem is more general than just improving the output of LLM coding tools

Yeah it’s extremely helpful to clarify your thoughts before starting work with LLM agents. I find Claude Code style plan mode to be a bit restrictive for me personally, but I’ve found that creating a plan doc and then collaboratively iterating on it with an LLM to be helpful here. I don’t really find it much different than the scoping I’d need to do before handing off some work to a more junior engineer.

> Claude Code style plan mode to be a bit restrictive

Hey thats why i built plannotator: https://github.com/backnotprop/plannotator

I like staying within Claude Code for orchestrating its plan mode, but I needed a better way to actually review the plan, address certain parts, see plan diffs, etc all in a better visual way. The hooks system through permissionrequest:exitplanmode keep this fairly ergonomic.

see it in action: https://www.youtube.com/watch?v=a_AT7cEN_9I

Re: LLMs work best when the user defines their acceptance criteria first

#353

Post by jghn »

_0ffh wrote: post #351
jghn wrote: post #329

Earlier quoted context omitted.

I think I see what you're saying and if so we're talking past each other a bit and I agree with what you're saying as well. The point I was raising is by the time an IC developer sees something, there's already been a process of curation that happens that frames the possible solutions & constrains branch points. This is different from saying that an IC makes 0 implementation decisions. The C-suite has set a direction…

I'm on board with that framing of the process, and I see how my original formulation was too rough. I was reacting to "We're moving to a world where the mechanical part of grinding the code is not worth much". I have the impression that in the past just mechanically grinding the code was less of a thing than it apparently is today. Guidance, sure, but not as much as seems to be common (often necessarily so) today. Bu…

Exactly. I was channeling the stereotypical dev that says they "just want to write code". To your point they're not literally *only* writing code, but this was the sort of person/mentality I was calling out.

What it says to me is they've actively avoided what appears to be becoming the most important skills in the new world. They're likely to find themselves on the short end of the stick.

Re: LLMs work best when the user defines their acceptance criteria first

#354

Post by plandis »

I’ve found this to be critical for having any chance of getting agents to generate code that is actually usable.

The more frequently you can verify correctness in some automated way the more likely the overall solution will be correct.

I’ve found that with good enough acceptance criteria (both positive and negative) it’s usually sufficient for agents to complete one off tasks without a human making a lot of changes. Essentially, if you’re willing to give up maintainability and other related properties, this works fairly well.

I’ve yet to find agents good enough to generate code that needs to be maintained long term without a ton of human feedback or manual code changes.

Re: LLMs work best when the user defines their acceptance criteria first

#355

Post by HarHarVeryFunny »

jmull wrote: post #230
D-Machine wrote: post #90

This article is great. And the blog-article headline is interesting, but wrong. LLM's don't in general write plausible code (as a rule) either. They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR. This isn't that hard to remember, and is a correct enough simplification of what generative LLMs actually do, without resorting…

> They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR. "Plausible" sounds like the right word to me. (It would be a mistake to digress into these features of LLMs in an article where it isn't needed.)

I agree - I took "plausible" here to mean plausible-looking, no different than similar-looking.

The trouble of course is that similar/plausible isn't good enough unless the LLM has seen enough similar-but-different training samples to refine it's notion of similarity to the point where it captures the differences that are critical in a given case.

I'd rather just characterize it as a lack of reasoning, since "add more data" can't be the solution to a world full of infinite variety. You can keep playing whack a mole to add more data to fix each failure, and I suppose it's an interesting experiment to see how far that will get you, but in the end the LLM is always going to be brittle and susceptible to stupid failure cases if it doesn't have the reasoning capability to fully analyze problems it was not trained on.

Re: LLMs work best when the user defines their acceptance criteria first

#356

Post by satvikpendem »

topaz0 wrote: post #337
satvikpendem wrote: post #334

Earlier quoted context omitted.

How do you know ours is any different, that we are not in a simulation or a solipsistic scenario? The truth is that one cannot know, it's a philosophical quandary that's been debated for millennia.

It is absolutely obvious how different it is from interacting with any LLM about the ways that it is wrong.

Nope, appeal to obviousness is not a sound argument. There are many things people thought were obvious that were wrong.

Re: LLMs work best when the user defines their acceptance criteria first

#357

Post by topaz0 »

satvikpendem wrote: post #356
topaz0 wrote: post #337

Earlier quoted context omitted.

It is absolutely obvious how different it is from interacting with any LLM about the ways that it is wrong.

Nope, appeal to obviousness is not a sound argument. There are many things people thought were obvious that were wrong.

It wasn't an argument. There isn't much point in going to a lot of trouble to make an argument to someone so clearly determined to ignore the truth. It is nevertheless true.

Re: LLMs work best when the user defines their acceptance criteria first

#358

Post by Implicated »

marginalia_nu wrote: post #201
Implicated wrote: post #91

Earlier quoted context omitted.

You're holding it wrong. Set the boundaries and guidelines before it starts working. Don't leave it space to do things you don't understand. ie: enforce conventions, set specific and measurable/verifiable goals, define skeletons of the resulting solutions if you want/can. To give an example. I do a lot of image similarity stuff and I wanted to test the Redis VectorSet stuff when it was still in beta and the PHP exten…

So in my experience with Opus 4.6 evaluating it in an existing code base has gone like this. You say "Do this thing". - It does the thing (takes 15 min). Looks incredibly fast. I couldn't code that fast. It's inhuman. So far all the fantastical claims hold up. But still. You ask "Did you do the thing?" - it says oops I forgot to do that sub-thing. (+5m) - it fixes the sub-thing (+10m) You say is the change well integ…

You didn't use plan mode.

Re: LLMs work best when the user defines their acceptance criteria first

#359

Post by Implicated »

ModernMech wrote: post #223
Implicated wrote: post #91

Earlier quoted context omitted.

You're holding it wrong. Set the boundaries and guidelines before it starts working. Don't leave it space to do things you don't understand. ie: enforce conventions, set specific and measurable/verifiable goals, define skeletons of the resulting solutions if you want/can. To give an example. I do a lot of image similarity stuff and I wanted to test the Redis VectorSet stuff when it was still in beta and the PHP exten…

Enforce conventions, be specific, and define boundaries… in English?!

Can you not? If not, learn how to. You'll find it helps immensely.

Re: LLMs work best when the user defines their acceptance criteria first

#360

Post by satvikpendem »

topaz0 wrote: post #357
satvikpendem wrote: post #356

Earlier quoted context omitted.

Nope, appeal to obviousness is not a sound argument. There are many things people thought were obvious that were wrong.

It wasn't an argument. There isn't much point in going to a lot of trouble to make an argument to someone so clearly determined to ignore the truth. It is nevertheless true.

Just saying something is true doesn't make it so. Truth requires justification, and if you can't provide that, then there's no reason to believe it's true. For someone making a claim, the onus is on them to provide evidence.

Otherwise I'll just say I'm right and you're wrong, after all, that's what you're saying.

Post reply on HN