Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

51–60 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#51

Post by marginalia_nu »

zeroxfe wrote: post #43
marginalia_nu wrote: post #30

Earlier quoted context omitted.

I didn't provide any constraints on how to draw it. TBH I would have just rendered a font glyph, or failing that, grabbed an image. Drawing it with vector graphics programmatically is very hard, but a decent programmer would and should push back on that.

> TBH I would have just rendered a font glyph, or failing that, grabbed an image. If an LLM did that, people would be all up in arms about it cheating. :-) For all its flaws, we seem to hold LLMs up to an unreasonably high bar.

That's the job description for a good programmer though. Question assumptions and requirements, and then find the simplest solution that does the job.

Just about anyone can eventually come up with a hideously convoluted HeraldicImageryEngineImplFactory.

Re: LLMs work best when the user defines their acceptance criteria first

#52

Post by internet2000 »

marginalia_nu wrote: post #2

I tried to make Claude Code, Sonnet 4.6, write a program that draws a fleur-de-lis. No exaggeration it floundered for an hour before it started to look right. It's really not good at tasks it has not seen before.

I got Opus 4.6 to one shot it, took 5-ish mins. "Write me a python program that outputs an svg of a fleur-de-lis. Use freely available images to double check your work."

It basically just re-created the wikipedia article fleur-de-lis, which I'm not sure proves anything beyond "you have to know how to use LLMs"

Re: LLMs work best when the user defines their acceptance criteria first

#53

Post by retired_account »

jqpabc123 wrote: post #48

LLMs have no idea what "correct" means. Anything they happen to get "correct" is the result of probability applied to their large training database. Being wrong will always be not only possible but also likely any time you ask for something that is not well represented in it's training data. The user has no way to know if this is the case so they are basically flying blind and hoping for the best. Relying on an LLM f…

It’s a shame of bulk of that training data is likely 2010s blogspam that was poor quality to begin with.

Re: LLMs work best when the user defines their acceptance criteria first

#54

Post by mmaunder »

serious_angel wrote: post #32
oofbey wrote: post #27

Earlier quoted context omitted.

It’s easy to get AI to write bad code. Turns out you still need coding skills to get AI to write good code. But those who have figured it out can crank out working systems at a shocking pace.

I am sorry for asking, but... is there guide even on how to "figure it out"? Otherwise, how are you so sure about it?

That's actually a great question. Truth be told the best way right now is to grab Codex CLI or Claude CLI (I strongly prefer Codex, but Claude has its fans), and just start. Immediately. Then go hard for a few months and you'll develop the skills you need.

A few tips for a quickstart:

Give yourself permission to play.

Understand basic concepts like context window, compaction, tokens, chain of thought and reasoning, and so on. Use AI to teach you this stuff, and read every blog post OpenAI and Anthropic put out and research what you don't understand.

Pick a hard coding problem in Python or Typescript and take a leap of faith and ask the agent to code it for you.

My favorite phrase when planning is: "Don't change anything. Just tell me.". Save this as a tmux shortcut and use it at the end of every prompt when planning something out.

Use markdown .md docs to create a planning doc and keep chatting to the agent about it and have it update the plan until you're super happy, always using the magic phrase "Don't change anything. Just tell me." (I should get myself a patent on that little number. Best trick I know)

Every time you see an anti-AI post, just move on. It's lazy people making lazy assumptions. Approach agentic coding with a sense of love, excitement, optimism, and take massive leaps of faith and you'll be very very surprised at what you find.

Best of luck Serious Angel.

Re: LLMs work best when the user defines their acceptance criteria first

#55

Post by vannevar »

pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

I'd highly recommend working top down, getting it to outline a sane architecture before it starts coding. Then if one of the modules starts getting fouled up, start with a clean sheet context (for that module) incorporating any cautions or lessons learned from the bad experience. LLMs are not yet good at working and reworking the same code, for the reasons you outline. But they are pretty good at a "Groundhog Day" approach of going through the implementation process over and over until they get it right.

Re: LLMs work best when the user defines their acceptance criteria first

#56

Post by pornel »

serious_angel wrote: post #32
oofbey wrote: post #27

Earlier quoted context omitted.

It’s easy to get AI to write bad code. Turns out you still need coding skills to get AI to write good code. But those who have figured it out can crank out working systems at a shocking pace.

I am sorry for asking, but... is there guide even on how to "figure it out"? Otherwise, how are you so sure about it?

When a new technology emerges we typically see some people who embrace it and "figure it out".

Electronic synthesisers went from "it's a piano, but expensive and sounds worse" to every weird preset creating a whole new genre of electronic music.

So it seems plausible, like Claude's code, that our complaints about unmaintainable code are from trying to use it like a piano, and the rave kids will find a better use for it.

Re: LLMs work best when the user defines their acceptance criteria first

#57

Post by 2god3 »

jqpabc123 wrote: post #48

LLMs have no idea what "correct" means. Anything they happen to get "correct" is the result of probability applied to their large training database. Being wrong will always be not only possible but also likely any time you ask for something that is not well represented in it's training data. The user has no way to know if this is the case so they are basically flying blind and hoping for the best. Relying on an LLM f…

Aye. I wish more conversations would be more of this nature - in that we should start with basic propositions - e.g. the thing does not 'know' or 'understand' what correct is.

Re: LLMs work best when the user defines their acceptance criteria first

#58

Post by LarsDu88 »

jqpabc123 wrote: post #48

LLMs have no idea what "correct" means. Anything they happen to get "correct" is the result of probability applied to their large training database. Being wrong will always be not only possible but also likely any time you ask for something that is not well represented in it's training data. The user has no way to know if this is the case so they are basically flying blind and hoping for the best. Relying on an LLM f…

This is about to change very soon. Unlike many other domains (such as greenfield scientific discovery), most coding problems for which we can write tests and benchmarks are "verifiable domains".

This means an LLM can autogenerated millions of code problem prompts, attempt millions of solutions (both working and non-working), and from the working solutions, penalize answers that have poor performance. The resulting synthetic dataset can then be used as a finetuning dataset.

There are now reinforcement finetuning techniques that have not been incorporated into the existing slate of LLMs that will enable finetuning them for both plausibility AND performance with a lot of gray area (like readability, conciseness, etc) in between.

What we are observing now is just the tip of a very large iceberg.

Re: LLMs work best when the user defines their acceptance criteria first

#59

Post by 2god3 »

retired_account wrote: post #53
jqpabc123 wrote: post #48

LLMs have no idea what "correct" means. Anything they happen to get "correct" is the result of probability applied to their large training database. Being wrong will always be not only possible but also likely any time you ask for something that is not well represented in it's training data. The user has no way to know if this is the case so they are basically flying blind and hoping for the best. Relying on an LLM f…

It’s a shame of bulk of that training data is likely 2010s blogspam that was poor quality to begin with.

But isn't that a reflection of reality?

If you've made a significant investment in human capital, you're even more likely to protect it now and prevent posting valuable stuff on the web.

Re: LLMs work best when the user defines their acceptance criteria first

#60

Post by kccqzy »

mmaunder wrote: post #24

But my AI didn't do what your AI did. Cherry picked AI fail for upvotes. Which you’ll get plenty of here an on Reddit from those too lazy to go and take a look for themselves. Using Codex or Claude to write and optimize high performance code is a game changer. Try optimizing cuda using nsys, for example. It’ll blow your lazy little brain.

Yeah right. A LLM in the hands of a junior engineer produces a lot of code that looks like they are written by juniors. A LLM in the hands of a senior engineer produces code that looks like they are written by seniors. The difference is the quality of the prompt, as well as the human judgement to reject the LLM code and follow-up prompts to tell the LLM what to write instead.
Post reply on HN