Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

331–340 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#331

Post by Jweb_Guru »

darkwater wrote: post #174
Jweb_Guru wrote: post #121

Earlier quoted context omitted.

It's not reality. I'm really not a fan of the way that people excuse the really terrible code LLMs write by claiming that people write code just as bad. Even if that were true, it is not true that when you ask those people to do otherwise they simply pretend to have done it and forget you asked later.

> it is not true that when you ask those people to do otherwise they simply pretend to have done it and forget you asked later. I had a coworker that more or less exactly did that. You left a comment in a ticket about something extra to be done, he answered "yes sure" and after a few days proceeded to close the ticket without doing the thing you asked. Depending on the quantity of work you had at the moment, you migh…

You may have had one. It clearly made a pretty negative impression on you because you are still complaining about them years later. I find it pretty misanthropic when people ascribe this kind of antisocial behavior to all of their coworkers.

Re: LLMs work best when the user defines their acceptance criteria first

#332

Post by salawat »

satvikpendem wrote: post #307
tovej wrote: post #298

Earlier quoted context omitted.

Could you please cite these papers. If by AI you mean LLMs, that is not supported by what I know. If you mean a theoretical world-model-based AI, that's just a tautological statement.

https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2506.02996

Their world model is completely a byproduct of language though, not experience. Furthermore, they by deliberate design do not maintain any form of self-recognition or narrative tracking, which is the necessary substrate for developing validating experience. The world model of an LLM is still a map. Not the territory. Even though ours has some of the same qualities arguably, the identity we carry with us and our self-narrative are incredibly powerful in terms of allowing us to maintain alignment with the world as she is without munging it up quite as badly as LLM's seem prone to.

Re: LLMs work best when the user defines their acceptance criteria first

#333

Post by Terretta »

valicord wrote: post #330
jghn wrote: post #214

Earlier quoted context omitted.

> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up t…

I agree with your first paragraph but not the second one. In many cases it's easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English, have an LLM turn them into code, and then have to carefully review that code to see if I forgot some detail that changes everything.

> easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English

Yes, and for team or company code, "there's the problem".

Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria.

Virtually all teams — because virtually all PgMs, PjMs, TLs, and Devs — miscalculate this.

Easier for you, not better for team or firm.

• • •

FWIW, perpetuation of this problem isn't really a fault of culture or skill or education. It's largely thanks to "leadership" having no idea how to correctly incentivize what the outcome should holistically be, as they don't know enough to know what long-haul good looks like.

FWIW, you can make that easier for them by having the LLM derive your acceptance criteria into English (based not only on code but on your entire conversation+iteration history) and write that up, which you can read and correct, after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating.

Even if you refuse to do spec driven development, LLMs can do development-driven spec. You can review that, you must correct it, and then ... Change can come after more easily — thanks to that context.

Re: LLMs work best when the user defines their acceptance criteria first

#334

Post by satvikpendem »

salawat wrote: post #332
satvikpendem wrote: post #307

Earlier quoted context omitted.

https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2506.02996

Their world model is completely a byproduct of language though, not experience. Furthermore, they by deliberate design do not maintain any form of self-recognition or narrative tracking, which is the necessary substrate for developing validating experience. The world model of an LLM is still a map. Not the territory. Even though ours has some of the same qualities arguably, the identity we carry with us and our self-…

How do you know ours is any different, that we are not in a simulation or a solipsistic scenario? The truth is that one cannot know, it's a philosophical quandary that's been debated for millennia.

Re: LLMs work best when the user defines their acceptance criteria first

#336

Post by awinter-py »

empath75 wrote: post #222
unlikelytomato wrote: post #70

Earlier quoted context omitted.

This is why I'm confused when people say it isn't ready to replace most of the programmer workforce.

If you a) know what you are doing and b) know what an llm is capable of doing, c) can manage multiple llm agents at a time, you can be unbelievably productive. Those skills I think are less common than people assume. You need to be technical, have good communication skills, have big picture vision, be organized, etc. If you are a staff level engineer, you basically feel like you don’t need anyone else. OTOH i have be…

it's like that '11 rules for showrunning' doc where you need to operate at a level where you understand the product being made, and the people making it, and their capabilities, in order to make things come out well without touching them directly.

(https://okbjgm.weebly.com/uploads/3/1/5/0/31506003/11_laws_o...)

if you can do every job + parallelize + read fast, and you are only limited by the time it takes to type, claude is remarkable. I'm not superhuman in those ways but in the small domains where I am it has helped a lot; in other domains it has ramped me to 'working prototype' 10x faster than I could have alone, but the quality of output seems questionable and I'm not smart enough to improve it

Re: LLMs work best when the user defines their acceptance criteria first

#337

Post by topaz0 »

satvikpendem wrote: post #334
salawat wrote: post #332

Earlier quoted context omitted.

Their world model is completely a byproduct of language though, not experience. Furthermore, they by deliberate design do not maintain any form of self-recognition or narrative tracking, which is the necessary substrate for developing validating experience. The world model of an LLM is still a map. Not the territory. Even though ours has some of the same qualities arguably, the identity we carry with us and our self-…

How do you know ours is any different, that we are not in a simulation or a solipsistic scenario? The truth is that one cannot know, it's a philosophical quandary that's been debated for millennia.

It is absolutely obvious how different it is from interacting with any LLM about the ways that it is wrong.

Re: LLMs work best when the user defines their acceptance criteria first

#338

Post by darkwater »

Jweb_Guru wrote: post #331
darkwater wrote: post #174

Earlier quoted context omitted.

> it is not true that when you ask those people to do otherwise they simply pretend to have done it and forget you asked later. I had a coworker that more or less exactly did that. You left a comment in a ticket about something extra to be done, he answered "yes sure" and after a few days proceeded to close the ticket without doing the thing you asked. Depending on the quantity of work you had at the moment, you migh…

You may have had one . It clearly made a pretty negative impression on you because you are still complaining about them years later. I find it pretty misanthropic when people ascribe this kind of antisocial behavior to all of their coworkers.

It's still relatively recent. Anyway I'm not saying everyone is like this, absolutely (not even an important chunk), but they do exist. At the same time it's not true that current LLMs only write terrible code.

Re: LLMs work best when the user defines their acceptance criteria first

#339

Post by valicord »

Terretta wrote: post #333
valicord wrote: post #330

Earlier quoted context omitted.

I agree with your first paragraph but not the second one. In many cases it's easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English, have an LLM turn them into code, and then have to carefully review that code to see if I forgot some detail that changes everything.

> easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English Yes, and for team or company code, "there's the problem". Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to th…

> Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria.

I have a lot of context about the system/codebase inside my head. 99.9% of it is not relevant to the specific task I need to do this week. The 0.1% that is relevant to this task is not relevant to other tasks that I or my teammates will need to do next week.

You're suggesting that I write down this particular 0.1% in some markdown file so that LLM can write the code for me, instead of writing the code myself (which would have been faster). Chances are, nobody is going to touch that particular piece of code again for a long time. By the time they do, whatever I have written down is likely out of date, so the long term benefit of writing everything down disappears.

> after the countless little iterations you made since your head-spec wasn't as concrete as you imagined before you started iterating.

That's exactly the point. If I need to iterate on the spec anyway, why would I use an intermediary (LLM) instead of just writing the code myself?

Re: LLMs work best when the user defines their acceptance criteria first

#340

Post by topaz0 »

satvikpendem wrote: post #261

Oftentimes, plausible code is good enough, hence why people keep using AI to generate code. This is a distinction without a difference.

2 seconds to insert 100 rows in an empty database table is not "good enough" if you are doing anything that is worth doing.
Post reply on HN