Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agentic Engineering Patterns

simonwillison.net

Post reply on HN
Print view View original

171–180 of 341 posts

Re: Agentic Engineering Patterns

#171

Post by jihadjihad »

alkonaut wrote: post #167
jihadjihad wrote: post #158

I wish there was a little more color in the Testing and QA section. While I agree with this: > A comprehensive test suite is by far the most effective way to keep those features working. there is no mention at all about LLMs' tendency to write tautological tests--tests that pass because they are defined to pass. Or, tests that are not at all relevant or useful, and are ultimately noise in the codebase wasting cycles…

This seems it should be very easy to validate. Force the AI to make minimal changes to the code under test, which makes a single (or as few as possible) test fail as a result. If it can't make a test fail at all, it should be useless.

Agreed, and that's why I think adding some example prompts and ideas to the Testing section would be helpful. A vanilla-prompted LLM, in my experience, is very unreliable at adding tests that fail when the changes are reverted.

Many times I've observed that the tests added by the model simply pass as part of the changes, but still pass even when those changes are no longer applied.

Re: Agentic Engineering Patterns

#172

Post by ndriscoll »

alkonaut wrote: post #167
jihadjihad wrote: post #158

I wish there was a little more color in the Testing and QA section. While I agree with this: > A comprehensive test suite is by far the most effective way to keep those features working. there is no mention at all about LLMs' tendency to write tautological tests--tests that pass because they are defined to pass. Or, tests that are not at all relevant or useful, and are ultimately noise in the codebase wasting cycles…

This seems it should be very easy to validate. Force the AI to make minimal changes to the code under test, which makes a single (or as few as possible) test fail as a result. If it can't make a test fail at all, it should be useless.

This is essentially dual to the idea behind mutation testing, and should be trivial to do with a mutation testing framework in place (track whether a given test catches mutants, or more sophisticated: whether it catches the exact same mutants as some other test).

Re: Agentic Engineering Patterns

#173

Post by JHer »

lbreakjai wrote: post #109

We're going to do it again, aren't we? We're going to take something simple and sensible ("write tests first", "small composable modules", etc.), give it a fancy complicated name ("Behavior-Constrained Implementation Lifecycle pattern", "Boundary-Scoped Processing Constructs pattern", etc.), and create an entire industry of consultants and experts selling books and enterprise coaching around it, each swearing they ha…

If all I have todo is ask the thing what I want, where is all the great new software? Why isn't everyone running fully bespoke operating systems by now?

While I agree with the sentiment that we shouldn't make things more complicated by inventing fancy names, we also shouldn't pretend that software engineering has become super simple now. Building a great piece of software remains super hard to do and finding better techniques for it affords real study.

Your post is annoying me quite a bit because it's super unfair to the linked post. Simon Willison isn't trying to coin a new term, he's just trying to start a collection of useful patterns. "Agentic engineering" is just the obvious term for software engineering using agents. What would you call it, "just asking things"?

Re: Agentic Engineering Patterns

#174

Post by simonw »

slaye wrote: post #168

Simon, if you're reading this, I'd be really curious to hear your thoughts on how to effectively conduct code reviews in a world where "code is cheap". One of the biggest struggles I have on my team is coworkers straight up vibing parts of the code and not understanding or guiding the architecture of subsystems. Or at least, not writing code in a way that is meant to be understood by others. Then when I go through th…

This is genuinely one of the most interesting questions right now. I don't have solid answers yet, and I'm very keen to learn what people are finding works.

If you accelerate the pace of code creation it inevitably creates bottlenecks elsewhere. Code review is by far the biggest of those right now.

There may be an argument for leaning less on code review. When code is expensive to produce and is likely to stay in production for many years it's obviously important to review it very carefully. If code is cheap and can be inexpensively replaced maybe we can lower our review standards?

But I don't want to lower my standards! I want the code I'm producing with coding agents to be better than the code I would produce without them.

There are some aspects of code review that you cannot skimp on. Things like coding standards may not matter as much, but security review will never be optional.

I've recently been wondering what we can learn from security teams at large companies. Once you have dozens or hundreds of teams shipping features at the same time - teams with varying levels of experience - you can no longer trust those teams not to make mistakes. I expect that the same strategies used by security teams at Facebook/Google-scale organizations could now be relevant to smaller organizations where coding agents are responsible for increasing amounts of code.

Generally though I think this is very much an unsolved problem. I hope to document the effective patterns for this as they emerge.

Re: Agentic Engineering Patterns

#175

Post by andy_ppp »

simonw wrote: post #146
63stack wrote: post #140

Earlier quoted context omitted.

People are rushing to be the first one to coin something and hit it big. Imagine the amount of $$$ you could get for being an "expert ai consultant" in this space. There was already another attempt at agentic patterns earlier: https://agentic-patterns.com/ Absolute hot air garbage.

Which pieces of my writing are garbage?

They won't have a decent response, this is the Internet after all. I really enjoyed it thanks for writing it and I'll take a lot of it onboard. I think everyone will have their own software stack and AIs designed perfectly for them to do their work in the future.

Re: Agentic Engineering Patterns

#176

Post by charlieflowers »

jvidalv wrote: post #131
maciusr wrote: post #116

[flagged]

I work as a consultant so I navigate different codebases, old to new, typescript to javascript, massive to small, frontend only to full stack. Claude Code experience is massively different depending on the codebase. Good E2E strongly typed codebase? Can one shot any feature, some small QA, some polishing and it's usually good to ship. Plain javascript? Object oriented? Injection? Overall magic? Claude can work there…

We are going to start seeing that be the primary selection criterion. Pick a stack that agents are good at.

Re: Agentic Engineering Patterns

#178

Post by darkwater »

fnands wrote: post #166
darkwater wrote: post #70

Earlier quoted context omitted.

> My experience with what coding assistants are good for shifted from: > smart autocomplete -> targeted changes/additions -> full engineering Define "full engineering". Because if you say "full engineering" I would expect the agent to get some expected product output details as input and produce all by itself the right implementation for the context (i.e. company) it lives in.

I agree that "full engineering" was a bit broad. I should probably have said something like "agent-only coding"? I.e. the point where the agent writes all the code and you just verify.

The "you just verify" part can take indeed a lot of steering and hand-holding to get the right implementation for the current company/department/project context. Otherwise you might be just generating tech debt at scale.

Re: Agentic Engineering Patterns

#179

Post by simonw »

alkonaut wrote: post #167
jihadjihad wrote: post #158

I wish there was a little more color in the Testing and QA section. While I agree with this: > A comprehensive test suite is by far the most effective way to keep those features working. there is no mention at all about LLMs' tendency to write tautological tests--tests that pass because they are defined to pass. Or, tests that are not at all relevant or useful, and are ultimately noise in the codebase wasting cycles…

This seems it should be very easy to validate. Force the AI to make minimal changes to the code under test, which makes a single (or as few as possible) test fail as a result. If it can't make a test fail at all, it should be useless.

That's part of the reason I like red/green TDD - you make the agent show that the test fails before the implementation and passes afterwards.

It can still cheat, but it's less likely to cheat.

Re: Agentic Engineering Patterns

#180

Post by simonw »

jihadjihad wrote: post #171
alkonaut wrote: post #167

Earlier quoted context omitted.

This seems it should be very easy to validate. Force the AI to make minimal changes to the code under test, which makes a single (or as few as possible) test fail as a result. If it can't make a test fail at all, it should be useless.

Agreed, and that's why I think adding some example prompts and ideas to the Testing section would be helpful. A vanilla-prompted LLM, in my experience, is very unreliable at adding tests that fail when the changes are reverted. Many times I've observed that the tests added by the model simply pass as part of the changes, but still pass even when those changes are no longer applied.

I had an example in that section but it got picked apart by pedants (who had good points) so I removed it. I plan to add another soon. You can still see it in the changelog: https://simonwillison.net/guides/agentic-engineering-pattern...
Post reply on HN