Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

411–420 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#411

Post by tovej »

seanmcdirmid wrote: post #402
tovej wrote: post #386

Earlier quoted context omitted.

I write code to solve a problem. Not code that looks like it solves the problem if a non-technical client squints at it. And if you don't prove your code, do you not design at all then? Do you never draw state diagrams? Every design is an informal proof of the solution. Rarely I write formal proofs. Most of the time I write down enough for myself to be convinced that the desing solves the problem.

Yes, you can dedicate extra tokens to draw state diagrams, the LLM can actually do that, if you don't have it generating one or more design documents before you are writing code you are doing that wrong. I still don't get how that is different from what humans are doing. > Most of the time I write down enough for myself to be convinced that the desing solves the problem. Again, why do you assume we aren't doing the s…

You completely misunderstood what I wrote.

It was a response to you saying: "Im not going around trying to prove my code correct when I write it manually."

How did you manage to forget what you wrote previously?

Also, in this post you are now suddenly taking the exact opposite position, contradicting your previous point.

Re: LLMs work best when the user defines their acceptance criteria first

#412

Post by aryehof »

_0ffh wrote: post #245
jghn wrote: post #214

Earlier quoted context omitted.

> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up t…

> This is hyperbolic Maybe a bit, but unfortunately sometimes not so much. I recently had an LLM write a couple of transforms on a tree in Python. The node class just had "kind" and "children" defined, nothing else. The LLM added new attributes to use in the new node kinds (Python allows to just do "foo.bar=baz" to add one). Apparently it saw a lot of code doing that during training. I corrected the code by hand and…

> what happened that made "coding" grunt work

Modern human programming has devolved to nothing more than modeling problems and systems using lines of code, procedures, sub-routines and modules, utilizing a “hack it till it works”(tm) methodology.

Re: LLMs work best when the user defines their acceptance criteria first

#413

Post by cadamsdotcom »

This is a bit unfair - to generate a bunch of code but not give the model data/tools and direct it to optimize it; then compare it to the optimized work of thousands over decades.

Feels like an extremely high effort hit piece, even though I know it’s not.

Re: LLMs work best when the user defines their acceptance criteria first

#414

Post by graphememes »

jqpabc123 wrote: post #407
graphememes wrote: post #225

Earlier quoted context omitted.

no but you're a human and you're responsible for it, so it's on you you can make horrible images with photoshop that doesn't make photoshop bad

The key word here is *you*. Photoshop doesn't make anything --- *you* make the image horrible --- or not. Any results relate directly to *your* skill. A direct comparison to agentic AI is less than equitable. AI is supposedly able to provide skill --- which it often fails to do.

you talk to the llm bro, you are responsible for the outcome

Re: LLMs work best when the user defines their acceptance criteria first

#415

Post by bluepoint »

pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

Actually I get improvements when I ask two llms to simplify each other’s work repeatedly.

Re: LLMs work best when the user defines their acceptance criteria first

#416

Post by seanmcdirmid »

tovej wrote: post #411
seanmcdirmid wrote: post #402

Earlier quoted context omitted.

Yes, you can dedicate extra tokens to draw state diagrams, the LLM can actually do that, if you don't have it generating one or more design documents before you are writing code you are doing that wrong. I still don't get how that is different from what humans are doing. > Most of the time I write down enough for myself to be convinced that the desing solves the problem. Again, why do you assume we aren't doing the s…

You completely misunderstood what I wrote. It was a response to you saying: "Im not going around trying to prove my code correct when I write it manually." How did you manage to forget what you wrote previously? Also, in this post you are now suddenly taking the exact opposite position, contradicting your previous point.

I did not contradict my previous point. But now I’m confused in how you think we use LLMs to write code. You made it sound like we just get it todump out code without any process in between.

Re: LLMs work best when the user defines their acceptance criteria first

#417

Post by Implicated »

leptons wrote: post #409
Implicated wrote: post #364

Earlier quoted context omitted.

Right - it's not a big deal and it LITERALLY is amateur hour. But I did it. I wouldn't have done it prior, sure I could have done a bunch of google searches but the time investment it would have taken to sift through all that information and distill it into actionable chunks would have far exceeded the benefit of doing so, in this case. The whole point is that it is amateur hour and it's wildly effective as a learnin…

>The whole point is that it is amateur hour and it's wildly effective as a learning tool. You sound so proud of your accomplishment, and I question if there's really nothing to be proud of here. I doubt you really learned anything , a machine told you what to do and you did it, like coloring by numbers - it doesn't make you an artist. You won't be able to build upon it, without asking the machine to do more of the th…

lmao - I'm not at all proud of what you called an accomplishment. I literally said it _is_ amateur hour, it's hacked together, not safe, not stylish, not well engineered. But it does work. And despite your assumption about me learning anything - I had _no idea_ how generators worked. The realization that spinning an electric motor would result in electricity being produced blew my mind and got me asking claude things related to that, then I wanted to interface a wheel against my wheel to spin a stepper motor to get a charge and had the hair brain idea to just make the whole thing the generator instead. None of this was stuff I knew.

Despite this thing I made being rather useless in the grand scheme of things it was _wildly_ illuminating in terms of my understanding of electricity and the various objects around me and how they function. Which has spurred another rabbit hole that is having _real measurable effect_ for a host of feral cats to live a more comfortable life. (Not the wheel generator thing)

> a machine told you what to do and you did it, like coloring by numbers - it doesn't make you an artist.

I never claimed to be an artist ;) And, maybe it's different for you, but someone or something showing me how to do something is quite literally the best way for me to learn. /shrug

> I have to doubt that. If you were all those things, you would have been able to complete that project with very little effort, and without a machine telling you what to do.

I love that for you.

Re: LLMs work best when the user defines their acceptance criteria first

#418

Post by treetalker »

otterley wrote: post #383
treetalker wrote: post #244

This is my experience with how LLMs "draft" legal arguments: at first glance, it's plausible — but may be, and often is, invalid, unsound, and/or ill-advised. The catch is that many judges lack the time, energy, or willingness to not only read the documents in detail, but also roll up their sleeves and dig into the arguments and cited authorities. (Some lack the skills, but those are extreme cases.) So the plausible…

As an attorney, I’m interested in this theory. Do you have any examples that illustrate the phenomenon you describe?

Sure, but which part: opposing counsel using LLMs; opposing counsel simply using bullshit asymmetry to befuddle (nothing new); or judges not always reading and looking deeply into the arguments and authorities (also nothing new)?

If the first category, there have been plenty of examples that have even made their way onto the HN front page in the last half year or so. There have even been instances of judges using LLMs to draft orders containing confabulated authorities.

Re: LLMs work best when the user defines their acceptance criteria first

#419

Post by theshrike79 »

_0ffh wrote: post #202
Foobar8568 wrote: post #113

Earlier quoted context omitted.

LLM code is higher quality than any codes I have seen in my 20 years in F500. So yeah you need to "guide" it, and ensure that it will not bypass all the security guidance for ex...But at least you are in control, although the cognitive load is much higher as well than just "blind trust of what is delivered". But I can see the carnage with offshoring+LLM, or "most employees", including so call software engineer + LLM.

Huh, that explains a lot about the F500, and their buzzword slogans like "culture of excellence". LLM code is still mostly absurdly bad, unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. Edit: I'll admit though that the detailed explanation is often still much less work than typing everything yourself. But i…

If you're using words in English to "tell it in painstaking detail", you're doing it the hard way.

You can provide it with a tool to do that. Agents run tools in a loop, give it good tools. We have linters, code analysers, fuzzers and everything else.

Configure them correctly, tell the agent to use them (in painstaking detail) and it can't mess things up.

Re: LLMs work best when the user defines their acceptance criteria first

#420

Post by theshrike79 »

valicord wrote: post #339
Terretta wrote: post #333

Earlier quoted context omitted.

> easier for me to directly write the code that satisfies the unwritten acceptance criteria I have in my head than to write those criteria down in English Yes, and for team or company code, "there's the problem". Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to th…

> Those acceptance criteria are guardrails for the change that comes after, and getting those out of your head into English is more important over the long haul than your undocumented short-term solution to the criteria. I have a lot of context about the system/codebase inside my head. 99.9% of it is not relevant to the specific task I need to do this week. The 0.1% that is relevant to this task is not relevant to ot…

You also write it down so that the bus factor goes down.

And if nobody touches the code for a long time, how can the documentation be out of date?

Post reply on HN