LLMs work best when the user defines their acceptance criteria first • Hacker News Board

Re: LLMs work best when the user defines their acceptance criteria first

#421

Post by theshrike79 » Sun, Mar 08, 2026, 8:50 PM UTC

marginalia_nu wrote: post #76

pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

My sense is that the code generation is fast, but then you always need to spend several hours making sure the implementation is appropriate, correct, well tested, based on correct assumptions, and doesn't introduce technical debt. You need to do this when coding manually as well, but the speed at which AI tools can output bad code means it's so much more important.

"Several hours"? How big are your change sets?

If a human dropped a PR on me that took "several hours" to go through (10k+ lines or non-trivial changes), I'd jump in my car and drive to the office just to specifically slap them on the back of the head ffs.

Re: LLMs work best when the user defines their acceptance criteria first

#422

Post by theshrike79 » Sun, Mar 08, 2026, 8:55 PM UTC

leptons wrote: post #409

Implicated wrote: post #364

Earlier quoted context omitted.

Right - it's not a big deal and it LITERALLY is amateur hour. But I did it. I wouldn't have done it prior, sure I could have done a bunch of google searches but the time investment it would have taken to sift through all that information and distill it into actionable chunks would have far exceeded the benefit of doing so, in this case. The whole point is that it is amateur hour and it's wildly effective as a learnin…

>The whole point is that it is amateur hour and it's wildly effective as a learning tool. You sound so proud of your accomplishment, and I question if there's really nothing to be proud of here. I doubt you really learned anything , a machine told you what to do and you did it, like coloring by numbers - it doesn't make you an artist. You won't be able to build upon it, without asking the machine to do more of the th…

What would be the appropriate way to learn then?

If a human gave me the same "amateur hour" instructions, would that be bad?

If I follow a "make exercise wheel display RPM" tutorial on a website, will I learn?

If it's in a book (distilled information is bad, right?), will I learn then?

Re: LLMs work best when the user defines their acceptance criteria first

#423

Post by theshrike79 » Sun, Mar 08, 2026, 8:58 PM UTC

roarcher wrote: post #303

treetalker wrote: post #244

This is my experience with how LLMs "draft" legal arguments: at first glance, it's plausible — but may be, and often is, invalid, unsound, and/or ill-advised. The catch is that many judges lack the time, energy, or willingness to not only read the documents in detail, but also roll up their sleeves and dig into the arguments and cited authorities. (Some lack the skills, but those are extreme cases.) So the plausible…

> LLM use in litigation drafting is thus akin to insurgent/guerilla warfare: it take little time, energy, or thinking to create, yet orders of magnitude more to analyze and refute. The same goes for coding. I have coworkers who use it to generate entire PRs. They can crank out two thousand lines of code that includes tests "proving" that it works, but may or may not actually be nonsense, in minutes. And then some poo…

If code smells like LLM, then you walk to said coworker and ask them to explain it for you. Play dumb if necessary.

Or you use YOUR LLM to review the PR :D

...and wtf, you get "credited" story points for finishing tasks? That sounds completely insane.

Re: LLMs work best when the user defines their acceptance criteria first

#424

Post by theshrike79 » Sun, Mar 08, 2026, 9:02 PM UTC

grey-area wrote: post #157

This is a fascinating look into code generated by an LLM that is correct in one sense (passes tests) but doesn't meet requirements (painfully slow). Doesn't use is_ipk to identify primary keys, uses fsync on every statement. The problem with larger projects like this even if you are competent is that there are just too many lines of code to read it properly and understand it all. Bravo to the author for taking the ti…

> This is a fascinating look into code generated by an LLM that is correct in one sense (passes tests) but doesn't meet requirements (painfully slow).

Why isn't requirements testing automated? Benchmarking the speed isn't rocket science. At worst a nightly build should run a benchmark and log it so you can find any anomalies.

Re: LLMs work best when the user defines their acceptance criteria first

#425

Post by theshrike79 » Sun, Mar 08, 2026, 9:03 PM UTC

grey-area wrote: post #199

g947o wrote: post #22

Earlier quoted context omitted.

Attributing these to "hidden requirements" is a slippery slope. My own experience using Claude Code and similar tools tells me that "hidden requirements" could include: * Make sure DESIGN.md is up to date * Write/update tests after changing source, and make sure they pass * Add integration test, not only unit tests that mock everything * Don't refactor code that is unrelated to the current task ... These are not even…

The training data is full of ‘any’ so you will keep getting ‘any’ because that is the code the models have seen. An interesting example of the training data overriding the context.

Then you add a biome rule to say "no any ever" and the LLM will fix it before claiming the job is done.

Re: LLMs work best when the user defines their acceptance criteria first

#426

Post by salawat » Sun, Mar 08, 2026, 10:03 PM UTC

satvikpendem wrote: post #334

salawat wrote: post #332

Earlier quoted context omitted.

Their world model is completely a byproduct of language though, not experience. Furthermore, they by deliberate design do not maintain any form of self-recognition or narrative tracking, which is the necessary substrate for developing validating experience. The world model of an LLM is still a map. Not the territory. Even though ours has some of the same qualities arguably, the identity we carry with us and our self-…

How do you know ours is any different, that we are not in a simulation or a solipsistic scenario? The truth is that one cannot know, it's a philosophical quandary that's been debated for millennia.

Simple. I have two sets of data I can pull from to validate a claim an LLM makes. I have the linguistic corpora we produce (artificial memory, analogical to latent space built by an LLM). You are correct in that this modality is shared. I also, however, have internal self-narrative and experiential state that is non-linguistic, but sensory/perception driven. An LLM can try to convince me that a bunch of mathematicians would come up with a system that requires one to make many copies of the same bitwise representation of a block for loading by the execution framework due to munging of the latent space via quantization. However, I have recollections of my time amongst Mathematicians and theorists. I can replay my lived perceptions of those times, and analyze and extract new meaning from them as my neural hardware evolves. Therefore, when that claim is made, my validation of the world as she is comes to a screeching halt to the tune of a recollection of a calculus class where the entire point is to pound into you the utility of fungibility of mathematical representations (substitution), and a further connection to optimization (replace entire cluster of an equation with a letter to process other things first and deal with the internal details later). That synthesizes also to the principle Mathematicians are both lazy, and clever. Alias that bitch, and moving right along. LLM's don't have that without you deliberately injecting that mechanism into their context. They'll in fact just run off the rails.

Now, could an equivalent process be modelled at some point? Probably. It'd be a conscious decision to do so on our part, and given fears over the AI Alignment quandary, it seems a rather fraught direction to carelessly proceed.

Re: LLMs work best when the user defines their acceptance criteria first

#427

Post by froggit » Sun, Mar 08, 2026, 11:03 PM UTC

aryehof wrote: post #412

_0ffh wrote: post #245

Earlier quoted context omitted.

> This is hyperbolic Maybe a bit, but unfortunately sometimes not so much. I recently had an LLM write a couple of transforms on a tree in Python. The node class just had "kind" and "children" defined, nothing else. The LLM added new attributes to use in the new node kinds (Python allows to just do "foo.bar=baz" to add one). Apparently it saw a lot of code doing that during training. I corrected the code by hand and…

> what happened that made "coding" grunt work Modern human programming has devolved to nothing more than modeling problems and systems using lines of code, procedures, sub-routines and modules, utilizing a “hack it till it works”(tm) methodology.

> utilizing a “hack it till it works”(tm) methodology.

Your post describes my coding perfectly. I don't have CS training of any type, never been formally involved in software development (recently started dabbling in OSS) and never used an LLM/agent for help (do use a local SLM for autocomplete and suggestions only).

Yet I can "code." I suspect a (pre-2023ish) software developer would likely tell me "go learn to code" if i asked for review. I don't know the formal syntax people expect to see and it has organization more typical of raging dumpster fires. Doesn't mean it's not code.

Re: LLMs work best when the user defines their acceptance criteria first

#428

Post by thaumasiotes » Mon, Mar 09, 2026, 12:52 AM UTC

thaumasiotes wrote: post #167

Implicated wrote: post #123

Earlier quoted context omitted.

You'll find, at times, that those communicating in a language that's not their primary language will tend to deviate from what one whose it was their primary language might expect. If that's obvious to you than you're just being rude. If it's not obvious to you, then you'll also find this is a common deviance (plural 'code') from those who come from a particular primary language's region. Edit; This got me thinking -…

> what is the grammar/rule around what gets pluralized and what doesn't? How does one know that "code" can refer to a single line of code, a whole file of code, a project, or even the entirety of all code your eyes have ever seen without having to have an s tacked on to the end of it? Well, the grammar is that English has two different classes of noun, and any given noun belongs to one class or the other. Standard te…

For the sake of completeness, I should mention that mass nouns, as a matter of grammar, do not and cannot have plural forms.

Re: LLMs work best when the user defines their acceptance criteria first

#429

Post by [deleted] » Mon, Mar 09, 2026, 1:16 AM UTC

theshrike79 wrote: post #423

roarcher wrote: post #303

Earlier quoted context omitted.

> LLM use in litigation drafting is thus akin to insurgent/guerilla warfare: it take little time, energy, or thinking to create, yet orders of magnitude more to analyze and refute. The same goes for coding. I have coworkers who use it to generate entire PRs. They can crank out two thousand lines of code that includes tests "proving" that it works, but may or may not actually be nonsense, in minutes. And then some poo…

If code smells like LLM, then you walk to said coworker and ask them to explain it for you. Play dumb if necessary. Or you use YOUR LLM to review the PR :D ...and wtf, you get "credited" story points for finishing tasks? That sounds completely insane.

[deleted]

Re: LLMs work best when the user defines their acceptance criteria first

#430

Post by roarcher » Mon, Mar 09, 2026, 1:29 AM UTC

theshrike79 wrote: post #423

roarcher wrote: post #303

Earlier quoted context omitted.

> LLM use in litigation drafting is thus akin to insurgent/guerilla warfare: it take little time, energy, or thinking to create, yet orders of magnitude more to analyze and refute. The same goes for coding. I have coworkers who use it to generate entire PRs. They can crank out two thousand lines of code that includes tests "proving" that it works, but may or may not actually be nonsense, in minutes. And then some poo…

If code smells like LLM, then you walk to said coworker and ask them to explain it for you. Play dumb if necessary. Or you use YOUR LLM to review the PR :D ...and wtf, you get "credited" story points for finishing tasks? That sounds completely insane.

> you get "credited" story points for finishing tasks? That sounds completely insane.

Developers' names are attached to stories, and stories have points on them. Why is that insane, and how does your company track who did what?

I propose that the name on the story should be that of the reviewer since they did the work.