Earlier quoted context omitted.
with LLMs, tests cost nearly nothing of effort but provide tremendous value.
And you know those tests are correct how?
Look at what they are testing.
Agentic Engineering Patterns
251–260 of 341 posts
Re: Agentic Engineering Patterns
#252Simon, if you're reading this, I'd be really curious to hear your thoughts on how to effectively conduct code reviews in a world where "code is cheap". One of the biggest struggles I have on my team is coworkers straight up vibing parts of the code and not understanding or guiding the architecture of subsystems. Or at least, not writing code in a way that is meant to be understood by others. Then when I go through th…
Agent based code reviews is what you want. But you have to do set it up with really good context about what is wanted. You then review the reviews, keep improving the context it is working with. Make sure it's put into everyone's global context they work with as well. Weirdly this article doesn't really talk about the main agentic pattern - Plan (really important to start with a plan before code changes). iteratively…
I'm still trying to figure out how to write about planning.
The problem is Claude Code has a planning mode baked in, which works really well but is quite custom to how Claude Code likes to do things.
When I describe it as a pattern I want to stretch a little beyond the current default implementation in one of the most popular coding agents.
Re: Agentic Engineering Patterns
#253Simon, if you're reading this, I'd be really curious to hear your thoughts on how to effectively conduct code reviews in a world where "code is cheap". One of the biggest struggles I have on my team is coworkers straight up vibing parts of the code and not understanding or guiding the architecture of subsystems. Or at least, not writing code in a way that is meant to be understood by others. Then when I go through th…
Code review should be mandatory and reviewers should ask big PRs to be broken up, and its submitters to be able to defend every line of code. For when the computer is generating the code, the most important duty of the submitter is to vouch for it. To do otherwise creates the bad incentive of making others do all your QA, and nobody is going to be rewarded for that.
I just added a chapter which touches on that: https://simonwillison.net/guides/agentic-engineering-pattern...
Re: Agentic Engineering Patterns
#254I mainly work with documents as a white collar worker but have vibe coded a few bits. The thing I keep coming back to is that it's all code. Almost all white collar professions have at least some key outputs in code. Whether you are a store manager filling out reports or a marketing firm or a teacher, there is so much code. This means you can give claude code a branded document template, fill it out, include images e…
White colar work is just a lucky place to be, 99% of it is completely made up, there's people doing nothing, and people doing work of 10 people, does not matter, the work itself has no impact on anything.
A nice way to realize why this AI wave hasn't produced massive economy growth, it is mostly touching parts of economy which are parasitic and can't really create growth.
Re: Agentic Engineering Patterns
#255The "human in the loop at key checkpoints" pattern has been the most practically useful for us. We found that giving the agent full autonomy end-to-end produces subtly broken code that passes tests but violates implicit invariants you never thought to write down. Short loops with a human sanity check at decision forks catches that class of failure early. The thing I keep wrestling with is where exactly to place those…
[dead]
Re: Agentic Engineering Patterns
#256Earlier quoted context omitted.
Agent based code reviews is what you want. But you have to do set it up with really good context about what is wanted. You then review the reviews, keep improving the context it is working with. Make sure it's put into everyone's global context they work with as well. Weirdly this article doesn't really talk about the main agentic pattern - Plan (really important to start with a plan before code changes). iteratively…
I'm still trying to figure out how to write about planning. The problem is Claude Code has a planning mode baked in, which works really well but is quite custom to how Claude Code likes to do things. When I describe it as a pattern I want to stretch a little beyond the current default implementation in one of the most popular coding agents.
You could have a look at: https://github.com/jurriaan/aico
It does 2 things that are very important, 1: reviewing should not be done last, but during the process and 2: plans should result into verifyable specs, preferably in a natural language so you can avoid locking yourself into specific implementation details (the "how") too early.
Re: Agentic Engineering Patterns
#257Earlier quoted context omitted.
Agent based code reviews is what you want. But you have to do set it up with really good context about what is wanted. You then review the reviews, keep improving the context it is working with. Make sure it's put into everyone's global context they work with as well. Weirdly this article doesn't really talk about the main agentic pattern - Plan (really important to start with a plan before code changes). iteratively…
I'm still trying to figure out how to write about planning. The problem is Claude Code has a planning mode baked in, which works really well but is quite custom to how Claude Code likes to do things. When I describe it as a pattern I want to stretch a little beyond the current default implementation in one of the most popular coding agents.
Maintaining a high-quality requirements / specification document for large features prior to implementation, and then referencing it in "plan mode" prompts, feels like consensus best practice at this stage.
However a thing I'm finding quite valuable in my own workflows, but haven't seen much discussion of, is spending meaningful time with AI doing meta-planning of that document. For example, I'll spend many sessions partnered with AI just iterating on the draft document, asking it to think through details, play contrarian, surface alternatives, poke holes, identify points of confusion, etc. It's been so helpful for rapidly exploring a design space, and I frequently find it makes suggestions that are genuinely surprising or change my perspective about what we should build.
I feel like I know we're "done" when I thoroughly understand it, a fresh AI instance seems to really understand it (as evaluated by interrogating it), and neither of us can find anything meaningful to improve. At that point we move to implementation, and the actual code writing falls out pretty seamlessly. Plus, there's a high quality requirements document as a long-lived artifact.
Obviously this is a heavyweight process, but is suited for my domain and work.
ETA one additional practice: if the agent gets confused during implementation or otherwise, I find it's almost always due to a latent confusion about the requirements. Ask the agent why it did a thing, figure out how to clarify in the requirements, and try again from the top rather than putting effort into steering the current session.
Re: Agentic Engineering Patterns
#258We're going to do it again, aren't we? We're going to take something simple and sensible ("write tests first", "small composable modules", etc.), give it a fancy complicated name ("Behavior-Constrained Implementation Lifecycle pattern", "Boundary-Scoped Processing Constructs pattern", etc.), and create an entire industry of consultants and experts selling books and enterprise coaching around it, each swearing they ha…
Wait: "write tests first" isn't simple and it's controversial. The benefits of TDD in pure-human development are debatable (I'd argue, in many cases, even dubious). But the equation changes with LLMs, because the cost of generating tests (and of keeping them up to date) plummets, and test cases are some of the easiest code to generate and reason about.
It's not as simple an observation as you're making it out to be.
Re: Agentic Engineering Patterns
#259have loved simon wilson for a long long time and still do.
These patterns are all out of date by at least a year - the best devs I know were using Claude 3.5 like this
Re: Agentic Engineering Patterns
#260have loved simon wilson for a long long time and still do. These patterns are all out of date by at least a year - the best devs I know were using Claude 3.5 like this
Good. If they worked with Sonnet 3.5 a year ago that means they have sticking power and are worth writing about today.