LLMs work best when the user defines their acceptance criteria first • Hacker News Board

Re: LLMs work best when the user defines their acceptance criteria first

#211

Post by bryanrasmussen » Sat, Mar 07, 2026, 12:42 PM UTC

ashdksnndck wrote: post #182

bryanrasmussen wrote: post #62

Earlier quoted context omitted.

maybe there should be an LLM trained on a corpus of a deletions and cleanup of code.

I think this is in the training data since they use commit data from repos, but I imagine code deletions are rarer than they should be in the real data as well.

deleting and code cleanup is perhaps more an expression of seniority, and personal preferences. Maybe there should be the same kind style transfer with code that you see with graphical generative AI, "rewrite this code path in the style of Donald Knuth"

Re: LLMs work best when the user defines their acceptance criteria first

#212

Post by true_religion » Sat, Mar 07, 2026, 12:42 PM UTC

iLoveOncall wrote: post #153

Implicated wrote: post #129

Earlier quoted context omitted.

I got curious and had to fire up the ol LLM to find out what the story is about the words that aren't pluralized - TIL about countable and uncountable nouns. I wonder if the guy giving you trouble about your English speaks French.

I'm native French and nobody would consider code countable. "codes" makes no sense. We'd talk about "lines of code" as a countable in French just like in English.

Codes is a proper grammatical word in English, but we don’t use it in reference to general computer programming.

You can for example have two different organizations with different codes of conduct.

There is though nothing technically wrong with seeing each line of code as an complete individual code and referring to then multiple of them as codes.

Re: LLMs work best when the user defines their acceptance criteria first

#213

Post by 2god3 » Sat, Mar 07, 2026, 1:20 PM UTC

ehnto wrote: post #116

ozozozd wrote: post #92

Earlier quoted context omitted.

Exactly. It’s also easy to find yourself in the out-of-distribution territory. Just ask for some tree-sitter queries and watch Gemini 3, Opus 4.5 and GLM 5 hallucinate new directives.

I think this could be the key difference in how people are experiencing the tools. Using Claude in industries full of proprietary code is a totally different experience to writing some React components, or framework code in C#, PHP or Java. It's shockingly good at the later, but as you get into proprietary frameworks or newer problem domains it feels like AI in 2023 again, even with the benefit of the agentic harness…

You’ve hit the nail on the head.

I characterise llm’s as being black boxes that are filled with a dense pool of digital resources - that with the correct prompt you can draw out a mix of resources to produce an output.

But if the mix of resources you need isn’t there - it won’t work. This isn’t limited to just text. This also applies with video models - llms work better for prompts in which you are trying to get material that is widely available on the internet.

Re: LLMs work best when the user defines their acceptance criteria first

#214

Post by jghn » Sat, Mar 07, 2026, 1:23 PM UTC

_0ffh wrote: post #202

Foobar8568 wrote: post #113

Earlier quoted context omitted.

LLM code is higher quality than any codes I have seen in my 20 years in F500. So yeah you need to "guide" it, and ensure that it will not bypass all the security guidance for ex...But at least you are in control, although the cognitive load is much higher as well than just "blind trust of what is delivered". But I can see the carnage with offshoring+LLM, or "most employees", including so call software engineer + LLM.

Huh, that explains a lot about the F500, and their buzzword slogans like "culture of excellence". LLM code is still mostly absurdly bad, unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. Edit: I'll admit though that the detailed explanation is often still much less work than typing everything yourself. But i…

> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class.

This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up the guard rails, define the acceptance criteria, these are the people who get great results. The people who walk up and say "sudo make me a sandwich" do not.

Also the latter group complains that they don't see the point of the first group. Why would they put in all the work when they could just code? But what they don't see is that *someone* was always doing that work, it just wasn't them in the past. We're moving to a world where the mechanical part of grinding the code is not worth much, people who defined their existence as avoiding all the legwork will be left in the cold.

Re: LLMs work best when the user defines their acceptance criteria first

#215

Post by 2god3 » Sat, Mar 07, 2026, 1:24 PM UTC

oofbey wrote: post #110

2god3 wrote: post #61

Earlier quoted context omitted.

You're not really answering the question are you? Your answer is to play with it. Cool. But why cant you and others put together a proper guide lol? It cant be that hard. Go ahead and do it - it'll challenge the Anti-AI posters you are referencing. I and others want to see that debate.

Here’s some practical tips: Start small. Figure out what it (whatever tool you’re using) can do reliably at a quality level you’re comfortable with. Try other tools. There are tons. If it doesn’t get it right with the first prompt, iterate. Refine. Keep at it until you get there. When you have seen some pattern work, do that a bunch. It won’t always work. Write rules / prompts / skills to try to get it to avoid makin…

I’m not a SWE by trade so I could care less about your last comment.

But again this is all… vague. I’m personally not convinced at all.

I’ll be hiring for a large project soon, so I’ll see for myself what benefits (well I care about net benefits) these tools are providing in the workplace.

Re: LLMs work best when the user defines their acceptance criteria first

#216

Post by simonw » Sat, Mar 07, 2026, 1:32 PM UTC

consumer451 wrote: post #172

Nitpick/question: the "LLM" is what you get via raw API call, correct? If you are using an LLM via a harness like claude.ai, chatgpt.com, Claude Code, Windsurf, Cursor, Excel Claude plug-in, etc... then you are not using an LLM, you are using something more, correct? An example I keep hearing is "LLMs have no memory/understanding of time so ___" - but, agents have various levels of memory. I keep trying to explain th…

I like to use the term "coding agents" for LLM harnesses that have the ability to directly execute code.

This is an important distinction because if they can execute the code they can test it themselves and iterate on it until it works.

The ChatGPT and Claude chatbot consumer apps do actually have this ability now so they technically class as "coding agents", but Claude Code and Codex CLI are more obvious examples as that's their key defining feature, not a hidden capability that many people haven't spotted yet.

Re: LLMs work best when the user defines their acceptance criteria first

#217

Post by pornel » Sat, Mar 07, 2026, 1:33 PM UTC

Implicated wrote: post #88

pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

My comment was a summary of the situation, not literal prompts I use. I absolutely realize the work needs to be adequately described and agents must be steered in the right direction. The results also vary greatly depending on the task and the model, so devs see different rates of success.

On non-trivial tasks (like adding a new index type to a db engine, not oneshotting a landing page) I find that the time and effort required to guide an LLM and review its work can exceed the effort of implementing the code myself. Figuring out exactly what to do and how to do it is the hard part of the task. I don't find LLMs helpful in that phase - their assessments and plans are shallow and naive. They can create todo lists that seemingly check off every box, but miss the forest for the trees (and it's an extra work for me to spot these problems).

Sometimes the obvious algorithm isn't the right one, or it turns out that the requirements were wrong. When I implement it myself, I have all the details in my head, so I can discover dead-ends and immediately backtrack. But when LLM is doing the implementation, it takes much more time to spot problems in the mountains of code, and even more effort to tell when it's a genuinely a wrong approach or merely poor execution.

If I feed it what I know before solving the problem myself, I just won't know all the gotchas yet myself. I can research the problem and think about it really hard in detail to give bulletproof guidance, but that's just programming without the typing.

And that's when the models actually behave sensibly. A lot of the time they go off the rails and I feel like a babysitter instructing them "no, don't eat the crayons!", and it's my skill issue for not knowing I must have "NO eating crayons" in AGENTS.md.

Re: LLMs work best when the user defines their acceptance criteria first

#218

Post by [deleted] » Sat, Mar 07, 2026, 1:37 PM UTC

pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

[deleted]

Re: LLMs work best when the user defines their acceptance criteria first

#219

Post by ben_w » Sat, Mar 07, 2026, 1:37 PM UTC

thesz wrote: post #120

Foobar8568 wrote: post #113

Earlier quoted context omitted.

LLM code is higher quality than any codes I have seen in my 20 years in F500. So yeah you need to "guide" it, and ensure that it will not bypass all the security guidance for ex...But at least you are in control, although the cognitive load is much higher as well than just "blind trust of what is delivered". But I can see the carnage with offshoring+LLM, or "most employees", including so call software engineer + LLM.

> LLM code is higher quality than any codes I have seen in my 20 years in F500. "Any codes"?

FWIW, I've noticed that scientists (native English speakers at least) will say "codes" rather "code". I don't know if this is universal or just specific domains (physics) nor if this is common or rare, but I've noticed it.

Re: LLMs work best when the user defines their acceptance criteria first

#220

Post by simonw » Sat, Mar 07, 2026, 1:37 PM UTC

tartoran wrote: post #49

comex wrote: post #34

Earlier quoted context omitted.

Supposedly the frontier LLMs are multimodal and trained on images as well, though I don't know how much that helps for tasks that don't use the native image input/output support. Whatever the cause, LLMs have gotten significantly better over time at generating SVGs of pelicans riding bicycles: https://simonwillison.net/tags/pelican-riding-a-bicycle/ But they're still not very good.

I have to admit I'm seeing this for the first time and am somewhat impressed by the results and even think they will get better with more training, why not... But are these multimodal LLMs still LLMs though? I mean, they're still LLMs but with a sidecar that does other things and the training of the image takes place outside the LLMs so in a way the LLMs still don't "know" anything about these images, they're just ge…

Some of the LLMs that can draw (bad) pelicans on bicycles are text-input-only LLMs.

The ones that have image input do tend to do better though, which I assume is because they have better "spatial awareness" as part of having been trained on images in addition to text.

I use the term vLLMs or vision LLMs to define LLMs that are multimodal for image and text input. I still don't have a great name for the ones that can also accept audio.

The pelican test requires SVG output because asking a multimodal output model like Gemini Flash Image (aka Nano Banana) to create an image is a different test entirely.