Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs work best when the user defines their acceptance criteria first

blog.katanaquant.com

Post reply on HN
Print view View original

231–240 of 460 posts

Re: LLMs work best when the user defines their acceptance criteria first

#231

Post by raw_anon_1111 »

datagobes wrote: post #194
raw_anon_1111 wrote: post #74

The difference for me recently Write a lambda that takes an S3 PUT event and inserts the rows of a comma separated file into a Postgres database. Naive implementation: download the file from s3 and do a bulk insert - it would have taken 20 minutes and what Claude did at first. I had to tell it to use the AWS sql extension to Postgres that will load a file directly from S3 into a table. It took 20 seconds. I treat cod…

Same pattern in data engineering generally. LLMs default to the obvious row-by-row or download-then-insert approach and you have to steer them toward the efficient path (COPY, bulk loaders, server-side imports). Once you name the right primitive, they execute it correctly, permissions and all, as you found. The deeper issue is that "efficient ingest" depends heavily on context that's implicit in your setup: file size…

Also with Redshift - split the file up before ingestion to equal the number of nodes or combine a lot of small files into larger files before putting them into S3 and/or use an Athena CTAS command to combine a lot of small files into one big file.

So in my other case, the whole thing was

Web crawler (internal customer website) using Playwrite -> S3 -> SNS -> SQS -> Lambda (embed with Bedrock) -> S3 Vector Store.

Similar to what you said, I ran into Bedrock embedding service limits. Then once I told it that, it knew how to adjust the lambda concurrency limits. Of course I had to tell it to also adjust the sqs poller so messages wouldn’t be backed up in flight, then go to the DLQ without ever being processed.

Re: LLMs work best when the user defines their acceptance criteria first

#232

Post by G3rn0ti »

2god3 wrote: post #64
kccqzy wrote: post #60

Earlier quoted context omitted.

Yeah right. A LLM in the hands of a junior engineer produces a lot of code that looks like they are written by juniors. A LLM in the hands of a senior engineer produces code that looks like they are written by seniors. The difference is the quality of the prompt, as well as the human judgement to reject the LLM code and follow-up prompts to tell the LLM what to write instead.

Lol what. The difference is that the senior... is a senior. Ask yourself what characteristics comprises a senior vs junior... You're glossing over so much stuff. Moreover, how does the Junior grow and become the senior with those characteristics, if their starting point is LLMs?

This. I really wonder how trainees are supposed to grow in an age where they are asked not to code themselves but guide a machine doing so.

Re: LLMs work best when the user defines their acceptance criteria first

#233

Post by teucris »

This article hits on an important point not easily discerned from the title:

Sometimes good software is good due to a long history of hard-earned wins.

AI can help you get to an implementation faster. But it cannot magically summon up a battle-hardened solution. That requires going through some battles.

Great software takes time.

Re: LLMs work best when the user defines their acceptance criteria first

#234

Post by raincole »

Foobar8568 wrote: post #125
thesz wrote: post #120

Earlier quoted context omitted.

> LLM code is higher quality than any codes I have seen in my 20 years in F500. "Any codes"?

At least my comment hasn't been reviewed or written by a LLM. And in my French brain, code or codebase is countable and not uncountable.

Quite sure they're not criticizing your grammar, but your substance.

Re: LLMs work best when the user defines their acceptance criteria first

#235

Post by raw_anon_1111 »

svpyk wrote: post #77
raw_anon_1111 wrote: post #74

The difference for me recently Write a lambda that takes an S3 PUT event and inserts the rows of a comma separated file into a Postgres database. Naive implementation: download the file from s3 and do a bulk insert - it would have taken 20 minutes and what Claude did at first. I had to tell it to use the AWS sql extension to Postgres that will load a file directly from S3 into a table. It took 20 seconds. I treat cod…

Unlike junior developers, llms can take detailed instructions and produce outstanding results at first shot a good number of times.

While I’m pro LLMs over junior developers. The other issue with LLMs is even the most junior developer will learn your business context over time.

In my case, in consulting (cloud + app dev), I just start the AGENTS.md file with a summary of the contract (the SOW), my architectural diagram and the transcript of my design review with the customer.

Re: LLMs work best when the user defines their acceptance criteria first

#236

Post by raincole »

Implicated wrote: post #88
pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

Not trying to be snarky, with all due respect... this is a skill issue. It's a tool. It's a wildly effective and capable tool. I don't know how or why I have such a wildly different experience than so many that describe their experiences in a similar manner... but... nearly every time I come to the same conclusion that the input determines the output. > If they implement something with a not-so-great approach, they'l…

> Do you tell your coworker "Hey, your code is slow" and expect great results? You ask it to benchmark the code and then you ask it how it might be optimized.

...Really? I think 'hey we have a lot of customers reporting the app is laggy when they do X, could you take a look' is a very reasonable thing to tell your coworker who implemented X.

Re: LLMs work best when the user defines their acceptance criteria first

#237

Post by jt2190 »

ollybrinkman wrote: post #227

This maps directly to the shift happening in API design for agent-to-agent communication. Traditional API contracts assume a human reads docs and writes code once. But when agents are calling agents, the "contract" needs to be machine-verifiable in real-time. The pattern I've seen work: explicit acceptance criteria in API responses themselves. Not just status codes, but structured metadata: "This response meets JSON…

Interesting, but couldn’t the agent be given access to tools that allow it to make those evaluations without having to modify the API responses? (Maybe I’m not visualizing “API” the same way you are.)

Re: LLMs work best when the user defines their acceptance criteria first

#240

Post by m3kw9 »

pornel wrote: post #36

Their default solution is to keep digging. It has a compounding effect of generating more and more code. If they implement something with a not-so-great approach, they'll keep adding workarounds or redundant code every time they run into limitations later. If you tell them the code is slow, they'll try to add optimized fast paths (more code), specialized routines (more code), custom data structures (even more code).…

Yes that’s my observation too. I have to be double careful the longer they run a task. They like to hack and patch stuff even when I tell it I don’t prefer it.
Post reply on HN