Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

221–230 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#221
post #27

Earlier quoted context omitted.

It’s easy to get AI to write bad code. Turns out you still need coding skills to get AI to write good code. But those who have figured it out can crank out working systems at a shocking pace.

I am sorry for asking, but... is there guide even on how to "figure it out"? Otherwise, how are you so sure about it?

I'm working on one here: https://simonwillison.net/guides/agentic-engineering-pattern...

Re: LLMs work best when the user defines their acceptance criteria first

#222
post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

This is why I'm confused when people say it isn't ready to replace most of the programmer workforce.

If you a) know what you are doing and b) know what an llm is capable of doing, c) can manage multiple llm agents at a time, you can be unbelievably productive. Those skills I think are less common than people assume.

You need to be technical, have good communication skills, have big picture vision, be organized, etc. If you are a staff level engineer, you basically feel like you don’t need anyone else.

OTOH i have been seeing even fairly technical engineering managers struggle because they can’t get the LLMs to execute because they don’t know how to ask it what to do.

Re: LLMs work best when the user defines their acceptance criteria first

#223

Earlier quoted context omitted.

And it’s slower to review because you didn’t do the hard part of understanding the code as it was being written.

You're holding it wrong. Set the boundaries and guidelines before it starts working. Don't leave it space to do things you don't understand. ie: enforce conventions, set specific and measurable/verifiable goals, define skeletons of the resulting solutions if you want/can. To give an example. I do a lot of image similarity stuff and I wanted to test the Redis VectorSet stuff when it was still in beta and the PHP exten…

Enforce conventions, be specific, and define boundaries… in English?!

Re: LLMs work best when the user defines their acceptance criteria first

#224
post #92

This article is great. And the blog-article headline is interesting, but wrong. LLM's don't in general write plausible code (as a rule) either. They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR. This isn't that hard to remember, and is a correct enough simplification of what generative LLMs actually do, without resorting…

Exactly. It’s also easy to find yourself in the out-of-distribution territory. Just ask for some tree-sitter queries and watch Gemini 3, Opus 4.5 and GLM 5 hallucinate new directives.

I think in the long term, if an LLM can’t use a tool, people won’t stop using LLM’s, they’ll stop using the tool.

We are building everything right now with LLM agents as a primary user in mind and one of our principles is “hallucination driven development”. If LLMs hallucinate an interface to your product regularly, that is a desire path and you should create that interface.

Re: LLMs work best when the user defines their acceptance criteria first

#225

bad input > bad output idk what to say, just because it's rust doesn't mean it's performant, or that you asked for it to be performant. yes, llms can produce bad code, they can also produce good code, just like people

yes, llms can produce bad code, they can also produce good code, just like people Over time, you develop a feel for which human coders tend to be consistently "good" or "bad". And you can eliminate the "bad". With an LLM, output quality is like a box of chocolates, you never know what you're going to get. It varies based on what you ask and what is in it's training data --- which you have no way to examine in advance…

no but you're a human and you're responsible for it, so it's on you

you can make horrible images with photoshop that doesn't make photoshop bad

Re: LLMs work best when the user defines their acceptance criteria first

#226
post #101

Earlier quoted context omitted.

> If you ask to unify the duplication, it'll say "No problem, here's a brand new metamock abstract adapter framework that has a superset of all feature sets, plus two new metamock drivers for the older and the newer code! Let me know if you want me to write tests for the new adapters." Nevermind the fact that it only migrated 3 out of 5 duplicated sections, and hasn’t deleted any now-dead code.

Sounds like my coworkers.

people also piss in rivers, yet dumping raw sewage by million m^3 in the same rivers is generally (less so in uk) frowned upon...

Re: LLMs work best when the user defines their acceptance criteria first

#227
This maps directly to the shift happening in API design for agent-to-agent communication.

Traditional API contracts assume a human reads docs and writes code once. But when agents are calling agents, the "contract" needs to be machine-verifiable in real-time.

The pattern I've seen work: explicit acceptance criteria in API responses themselves. Not just status codes, but structured metadata: "This response meets JSON Schema v2.1, latency was 180ms, data freshness is 3 seconds."

Lets the calling agent programmatically verify "did I get what I paid for?" without human intervention. The measurement problem becomes the automation problem.

Similar to how distributed systems moved from "hope it works" to explicit SLOs and circuit breakers. Agents need that, but at the individual request level.

Re: LLMs work best when the user defines their acceptance criteria first

#228

> SQLite is not primarily fast because it is written in C. Well.. that too, but it is fast because 26 years of profiling have identified which tradeoffs matter. Someone (with deep pockets to bear the token costs) should let Claude run for 26 months to have it optimize its Rust code base iteratively towards equal benchmarks. Would be an interesting experiment. The article points out the general issue when discussing L…

This would be a really interesting experiment.

I suspect performance is not the only problem with the codebase though.

Re: LLMs work best when the user defines their acceptance criteria first

#230

This article is great. And the blog-article headline is interesting, but wrong. LLM's don't in general write plausible code (as a rule) either. They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR. This isn't that hard to remember, and is a correct enough simplification of what generative LLMs actually do, without resorting…

> They just write code that is (semantically) similar to code (clusters) seen in its training data, and which haven't been fenced off by RLHF / RLVR.

"Plausible" sounds like the right word to me. (It would be a mistake to digress into these features of LLMs in an article where it isn't needed.)

Post reply on HN