Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

441–450 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#441

Post by empath75 »

jurgenburgen wrote: post #408
empath75 wrote: post #222

Earlier quoted context omitted.

If you a) know what you are doing and b) know what an llm is capable of doing, c) can manage multiple llm agents at a time, you can be unbelievably productive. Those skills I think are less common than people assume. You need to be technical, have good communication skills, have big picture vision, be organized, etc. If you are a staff level engineer, you basically feel like you don’t need anyone else. OTOH i have be…

> can manage multiple llm agents at a time How is that supposed to work? Humans are notoriously poor at multi-tasking. If you spend all day context switching between agents you’re going to have a bad time.

I deployed a whole fleet of agents working Jira tickets and PRs a couple of weeks ago. You manage them the way you manage people.

Re: LLMs work best when the user defines their acceptance criteria first

#442

Post by otterley »

treetalker wrote: post #418
otterley wrote: post #383

Earlier quoted context omitted.

As an attorney, I’m interested in this theory. Do you have any examples that illustrate the phenomenon you describe?

Sure, but which part: opposing counsel using LLMs; opposing counsel simply using bullshit asymmetry to befuddle (nothing new); or judges not always reading and looking deeply into the arguments and authorities (also nothing new)? If the first category, there have been plenty of examples that have even made their way onto the HN front page in the last half year or so. There have even been instances of judges using LLM…

> opposing counsel simply using bullshit asymmetry to befuddle (nothing new)

This one in particular.

Re: LLMs work best when the user defines their acceptance criteria first

#444

Post by marginalia_nu »

theshrike79 wrote: post #421
marginalia_nu wrote: post #76

Earlier quoted context omitted.

My sense is that the code generation is fast, but then you always need to spend several hours making sure the implementation is appropriate, correct, well tested, based on correct assumptions, and doesn't introduce technical debt. You need to do this when coding manually as well, but the speed at which AI tools can output bad code means it's so much more important.

"Several hours"? How big are your change sets? If a human dropped a PR on me that took "several hours" to go through (10k+ lines or non-trivial changes), I'd jump in my car and drive to the office just to specifically slap them on the back of the head ffs.

This was like 1K LOC? It's not the review that was slow, but the wrestling with the model to get the code to not suck.

Re: LLMs work best when the user defines their acceptance criteria first

#445

Post by [deleted] »

raw_anon_1111 wrote: post #231
datagobes wrote: post #194

Earlier quoted context omitted.

Same pattern in data engineering generally. LLMs default to the obvious row-by-row or download-then-insert approach and you have to steer them toward the efficient path (COPY, bulk loaders, server-side imports). Once you name the right primitive, they execute it correctly, permissions and all, as you found. The deeper issue is that "efficient ingest" depends heavily on context that's implicit in your setup: file size…

Also with Redshift - split the file up before ingestion to equal the number of nodes or combine a lot of small files into larger files before putting them into S3 and/or use an Athena CTAS command to combine a lot of small files into one big file. So in my other case, the whole thing was Web crawler (internal customer website) using Playwrite -> S3 -> SNS -> SQS -> Lambda (embed with Bedrock) -> S3 Vector Store. Simi…

[deleted]

Re: LLMs work best when the user defines their acceptance criteria first

#447

Post by leptons »

theshrike79 wrote: post #422
leptons wrote: post #409

Earlier quoted context omitted.

>The whole point is that it is amateur hour and it's wildly effective as a learning tool. You sound so proud of your accomplishment, and I question if there's really nothing to be proud of here. I doubt you really learned anything , a machine told you what to do and you did it, like coloring by numbers - it doesn't make you an artist. You won't be able to build upon it, without asking the machine to do more of the th…

What would be the appropriate way to learn then? If a human gave me the same "amateur hour" instructions, would that be bad? If I follow a "make exercise wheel display RPM" tutorial on a website, will I learn? If it's in a book (distilled information is bad, right?), will I learn then?

OP was writing how great the LLM is, and that he couldn't do this stuff as easily before LLMs. And that simply isn't true.

Instead of breaking down the task himself into achievable steps, the LLM did that "thinking" for him. This will inevitably lead to atrophy of the brain. If you don't exercise your brain, and let the tin-can tell you what to do, you're going to get pretty dull. It's well known that keeping your brain active, solving problems, will keep your mental abilities strong. Using LLMs is the opposite of that.

Re: LLMs work best when the user defines their acceptance criteria first

#448

Post by scotty79 »

riffraff wrote: post #126
Implicated wrote: post #88

Earlier quoted context omitted.

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

> Whew. Ok. You don't tell it the code is slow. Do you tell your coworker "Hey, your code is slow" and expect great results? Yes? Why don't you? They are capable people that just didn't notice something, id I notice some telemetry and tell them "hey this is slow" they are expected to understand the reason(s).

There was a 20+ person team of well paid, smart (mostly Java) programmers that dealt for months with slow application they were building, that everyone knew was slow. I nagged them for weeks to set up indexes even for small, 100 row tables. Once they did things started running orders of magnitude faster.

Your expectations for people (and LLMs) are way too high.

Re: LLMs work best when the user defines their acceptance criteria first

#449

Post by scotty79 »

roarcher wrote: post #430
theshrike79 wrote: post #423

Earlier quoted context omitted.

If code smells like LLM, then you walk to said coworker and ask them to explain it for you. Play dumb if necessary. Or you use YOUR LLM to review the PR :D ...and wtf, you get "credited" story points for finishing tasks? That sounds completely insane.

> you get "credited" story points for finishing tasks? That sounds completely insane. Developers' names are attached to stories, and stories have points on them. Why is that insane, and how does your company track who did what? I propose that the name on the story should be that of the reviewer since they did the work.

Because story points is a tool for the business to know when optimistically a thing could be done. Or more realistically get a decent "no sooner than" estimation of the task.

Using them for anything else, or by anyone else, like scoring the team or like here, individual contributors, is idiotic.

Post reply on HN