Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

251–260 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#251

Post by queenkjuul »

_0ffh wrote: post #245
jghn wrote: post #214

Earlier quoted context omitted.

> unless you tell it in painstaking detail what to do and what to avoid, and never ask it to do a bigger job at a time than a single function or very small class. This is hyperbolic, but the general sentiment is accurate enough, at least for now. I've noticed a bimodal distribution of quality when using these tools. The people who approach the LLM from the lens of a combo architect & PM, do all the leg work, set up t…

> This is hyperbolic Maybe a bit, but unfortunately sometimes not so much. I recently had an LLM write a couple of transforms on a tree in Python. The node class just had "kind" and "children" defined, nothing else. The LLM added new attributes to use in the new node kinds (Python allows to just do "foo.bar=baz" to add one). Apparently it saw a lot of code doing that during training. I corrected the code by hand and…

I'm with you, it's constantly doing stupid shit and ignoring instructions, and I've always been responsible for determining architecture and doing the "legwork." Unless the task is so small and well defined that it's less typing to tell the LLM (and clean up its output) then i may as well just do it myself

Re: LLMs work best when the user defines their acceptance criteria first

#252

Post by apical_dendrite »

seanmcdirmid wrote: post #247

Ok, I’ll bite: how is that different from humans?

The volume is different. Someone submitted a PR this week that was 3800 lines of shell script. Most of it was crap and none of it should have been in shell script. He's submitting PRs with thousands of lines of code every day. He has no idea how any of it actually works, and it completely overwhelms my ability to review.

Sure, he could have submitted a ill-considered 3800 line PR five years ago, but it would have taken him at least a week and there probably would have been opportunities to submit smaller chunks along the way or discuss the approach.

Re: LLMs work best when the user defines their acceptance criteria first

#253

Post by somewhereoutth »

seanmcdirmid wrote: post #247

Ok, I’ll bite: how is that different from humans?

Humans have a 'world model' beyond the syntax - for code, an idea of what the code should do and how it does it. Of course, some humans are better than others at this, they are recognized as good programmers.

Re: LLMs work best when the user defines their acceptance criteria first

#254

Post by dahart »

iLoveOncall wrote: post #153
Implicated wrote: post #129

Earlier quoted context omitted.

I got curious and had to fire up the ol LLM to find out what the story is about the words that aren't pluralized - TIL about countable and uncountable nouns. I wonder if the guy giving you trouble about your English speaks French.

I'm native French and nobody would consider code countable. "codes" makes no sense. We'd talk about "lines of code" as a countable in French just like in English.

Codes can be synonymous with codebases and is grammatically just fine, though probably not the most common usage.

Re: LLMs work best when the user defines their acceptance criteria first

#256

Post by bitwize »

You: Claude, do you know how to program?

Claude: No, but if you hum a few bars I can fake it!

Except "faking it" turns out to be good enough, especially if you can fake it at speed and get feedback as to whether it works. You can then just hillclimb your way to an acceptable solution.

Re: LLMs work best when the user defines their acceptance criteria first

#257

Post by satvikpendem »

wood_spirit wrote: post #250
seanmcdirmid wrote: post #247

Ok, I’ll bite: how is that different from humans?

It’s not. LLMs are just averaging their internet snapshot, after all. But people want an AI that is objective and right. HN is where people who know the distinction hang out, but it’s not what the layperson things they are getting when they use this miraculous super hyped tool that everybody is raving about?

By now, a few years after ChatGPT released, I don't think anyone is thinking AI is objective and right, all users have seen at least one instance of hallucination and simply being wrong.

Re: LLMs work best when the user defines their acceptance criteria first

#258

Post by satvikpendem »

apical_dendrite wrote: post #252
seanmcdirmid wrote: post #247

Ok, I’ll bite: how is that different from humans?

The volume is different. Someone submitted a PR this week that was 3800 lines of shell script. Most of it was crap and none of it should have been in shell script. He's submitting PRs with thousands of lines of code every day. He has no idea how any of it actually works, and it completely overwhelms my ability to review. Sure, he could have submitted a ill-considered 3800 line PR five years ago, but it would have tak…

Just block that user, that seems to be the way.

Re: LLMs work best when the user defines their acceptance criteria first

#259

Post by dahart »

thesz wrote: post #135
Implicated wrote: post #129

Earlier quoted context omitted.

I got curious and had to fire up the ol LLM to find out what the story is about the words that aren't pluralized - TIL about countable and uncountable nouns. I wonder if the guy giving you trouble about your English speaks French.

I speak Russian and some English, but the question was about universal quantification: author declares that LLMs generate code of better quality than "any codes" he seen in his career.

LLMs got their training data from somewhere. But maybe they’re good at percolating the good code to the top and filtering the bad code.

Re: LLMs work best when the user defines their acceptance criteria first

#260

Post by satvikpendem »

somewhereoutth wrote: post #253
seanmcdirmid wrote: post #247

Ok, I’ll bite: how is that different from humans?

Humans have a 'world model' beyond the syntax - for code, an idea of what the code should do and how it does it. Of course, some humans are better than others at this, they are recognized as good programmers .

Papers show that AI also has a world model, so I don't think that's the right distinction.
Post reply on HN