Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

141–150 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#141

Post by Implicated »

otabdeveloper4 wrote: post #119
Implicated wrote: post #88

Earlier quoted context omitted.

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

It is not a tool. It is an oracle. It can be a tool, for specific niche problems: summarization, extraction, source-to-source translation -- if post-trained properly. But that isn't what y'all are doing, you're engaging in "replace all the meatsacks AGI ftw" nonsense.

If I was on the "replace all the meatsacks AGI ftw" team then I would have referred to it as an oracle, by your own logic, wouldn't I have?

It's a tool. It's good for some things, not for others. Use the right tool for the job and know the job well enough to know which tools apply to which tasks.

More than anything it's a learning tool. It's also wildly effective at writing code, too. But, man... the things that it makes available to the curious mind are rather unreal.

I used it to help me turn a cat exercise wheel (think huge hamster wheel) into a generator that produces enough power to charge a battery that powers an ESP32 powered "CYD" touchscreen LCD that also utilizes a hall effect sensor to monitor, log and display the RPMs and "speed" (given we know the wheel circumference) in real time as well as historically.

I didn't know anything about all this stuff before I started. I didn't AGI myself here. I used a learning tool.

But keep up with your schtick if that's what you want to do.

Re: LLMs work best when the user defines their acceptance criteria first

#142

Post by mettamage »

Foobar8568 wrote: post #113
unlikelytomato wrote: post #70

Earlier quoted context omitted.

This is why I'm confused when people say it isn't ready to replace most of the programmer workforce.

LLM code is higher quality than any codes I have seen in my 20 years in F500. So yeah you need to "guide" it, and ensure that it will not bypass all the security guidance for ex...But at least you are in control, although the cognitive load is much higher as well than just "blind trust of what is delivered". But I can see the carnage with offshoring+LLM, or "most employees", including so call software engineer + LLM.

Giving it prompts of the Shannon project helps for security

Re: LLMs work best when the user defines their acceptance criteria first

#143

Post by parvardegr »

tartoran wrote: post #4
marginalia_nu wrote: post #2

I tried to make Claude Code, Sonnet 4.6, write a program that draws a fleur-de-lis. No exaggeration it floundered for an hour before it started to look right. It's really not good at tasks it has not seen before.

Have you tried describing to Claude what it is? The more the detail the better the result. At some point it does become easier to just do it yourself.

agreed with part that at some point it's better to just do it yourself but for sure they will get better and better

Re: LLMs work best when the user defines their acceptance criteria first

#144

Post by stevenhuang »

red75prime wrote: post #117
tartoran wrote: post #33

Earlier quoted context omitted.

LLMs piggyback on human knowledge encoded in all the texts they were trained on without understanding what they're doing. Humans would execute that code and validate it. From plausible it'd becomes hey, it does this and this is what I want. LLMs skip that part, they really have no understanding other than the statistical patterns they infer from their training and they really don't need any for what they are.

Could we stop using vague terms like “understanding” when talking about LLMs and machine learning? You don't know what understanding is. You only know how it feels to understand something. It's better to describe what you can do that LLMs currently can't.

At least it's an easy way for those who don't know that they're talking about to out themselves.

If they'd bother to see how modern neuroscience tries to explain human cognition they'd see it explained in terms that parallel modern ML. https://en.wikipedia.org/wiki/Predictive_coding

We only have theories for what intelligence even means, I wouldn't be surprised there are more similarities than differences between human minds and LLMs, fundamentally (prediction and error minimization)

Re: LLMs work best when the user defines their acceptance criteria first

#145

Post by coolius »

vannevar wrote: post #55
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

I'd highly recommend working top down, getting it to outline a sane architecture before it starts coding. Then if one of the modules starts getting fouled up, start with a clean sheet context (for that module) incorporating any cautions or lessons learned from the bad experience. LLMs are not yet good at working and reworking the same code, for the reasons you outline. But they are pretty good at a "Groundhog Day" ap…

+1 if you are vibe coding projects from scratch. if the architecture you specify doesn't make sense, the llm will start struggling, the only way out of their misery is mocking tests. the good thing is that a complete rewrite with proper architecture and lessons learned is now totally affordable.

Re: LLMs work best when the user defines their acceptance criteria first

#146

Post by 5o1ecist »

Implicated wrote: post #88
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

[dead]

Re: LLMs work best when the user defines their acceptance criteria first

#147

Post by einrealist »

> SQLite is not primarily fast because it is written in C. Well.. that too, but it is fast because 26 years of profiling have identified which tradeoffs matter.

Someone (with deep pockets to bear the token costs) should let Claude run for 26 months to have it optimize its Rust code base iteratively towards equal benchmarks. Would be an interesting experiment.

The article points out the general issue when discussing LLMs: audience and subject matter. We mostly discuss anecdotally about interactions and results. We really need much more data, more projects to succeed with LLMs or to fail with them - or to linger in a state of ignorance, sunk-cost fallacy and supressed resignation. I expect the latter will remain the standard case that we do not hear about - the part of the iceberg that is underwater, mostly existing within the corporate world or in private GitHubs, a case that is true with LLMs and without them.

In my experience, 'Senior Software Engineer' has NO general meaning. It's a title to be awarded for each participation in a project/product over and over again. The same goes for the claim: "Me, Senior SWE treat LLMs as Junior SWE, and I am 10x more productive." Imagine me facepalming every time.

Re: LLMs work best when the user defines their acceptance criteria first

#148

Post by sebastiennight »

Foobar8568 wrote: post #125
thesz wrote: post #120

Earlier quoted context omitted.

> LLM code is higher quality than any codes I have seen in my 20 years in F500. "Any codes"?

At least my comment hasn't been reviewed or written by a LLM. And in my French brain, code or codebase is countable and not uncountable.

As far as I've ever heard, "le code" used in a codebase is uncountable, like "le café" you'd put in a cup, so we would still say "meilleur que tout le code que j'ai vu en 20 ans" and not "meilleur que tous les codes que j'ai vus en 20 ans".

There is a countable "code" (just like "un café" is either a place, or a cup of coffee, or a type of coffee), and "un code" would be the one used as a password or secret, as in "j'ai utilisé tous les codes de récupération et perdu mon accès Gmail" (I used all the recovery codes and lost Gmail access).

Re: LLMs work best when the user defines their acceptance criteria first

#149

Post by bryanrasmussen »

riffraff wrote: post #126
Implicated wrote: post #88

Earlier quoted context omitted.

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

> Whew. Ok. You don't tell it the code is slow. Do you tell your coworker "Hey, your code is slow" and expect great results? Yes? Why don't you? They are capable people that just didn't notice something, id I notice some telemetry and tell them "hey this is slow" they are expected to understand the reason(s).

Yeah if my co-worker can't start figuring out why the code is slow, with a reasonable reference to what the code in question is, that is a knock against their skills. I would actually expect some ideas as to what the problem is just off the top of their heads, but that the coding agent can't do that isn't a hit against it specifically, this is now a good part of what needs to be done differently.

The suggestion to tell the agent to do performance analysis of the part of the code you think is problematic, and offer suggestions for improvements seems like the proper way to talk to a machine, whereas "hey your code is slow" feels like the proper way to talk to a human.

Re: LLMs work best when the user defines their acceptance criteria first

#150

Post by mejutoco »

thesz wrote: post #127
Foobar8568 wrote: post #125

Earlier quoted context omitted.

At least my comment hasn't been reviewed or written by a LLM. And in my French brain, code or codebase is countable and not uncountable.

I guess you can guide it to write in any style. But what set me off is an universal qualifier: there was no code seen by you that is of equal quality or better that what LLMs generate.

cows are brown, from one side.

https://www.neatorama.com/2007/01/22/a-mathematical-cow-joke...

Post reply on HN