Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agentic Engineering Patterns

simonwillison.net

Post reply on HN
Print view View original

291–300 of 341 posts

Re: Agentic Engineering Patterns

#291

Post by tveita »

tshaddox wrote: post #189
jihadjihad wrote: post #158

I wish there was a little more color in the Testing and QA section. While I agree with this: > A comprehensive test suite is by far the most effective way to keep those features working. there is no mention at all about LLMs' tendency to write tautological tests--tests that pass because they are defined to pass. Or, tests that are not at all relevant or useful, and are ultimately noise in the codebase wasting cycles…

Do you have an example of the tautological tests you're referring to? What comes to mind to me is genuinely logically tautological tests, like "assert(true || expectedResult == actualResult)" which is a mistake I don't even expect modern AI coding tools to make. But I suspect you're talking about a subtler type of test which at first glance appears useful but actually isn't.

I've definitely seen Opus go to town when asked to test a fairly simple builder. Possibly it inferred something about testing the "contract", and went on to test such properties as

  - none of the "final" fields have changed after calling each method
  - these two immutable objects we just confirmed differ on a property are not the same object
In addition to multiple tests with essentially identical code, multiple test classes with largely duplicated tests etc.

Re: Agentic Engineering Patterns

#292

Post by toraway »

sarkarsh wrote: post #216
MartyMcBot wrote: post #132

Earlier quoted context omitted.

[flagged]

[dead]

BTW, check the comment history of the above account @sarkash, this is almost certainly an LLM replying with the exact same structure/format in all their comments.

  This is the underrated insight in the whole thread
From comment history:

  This is good advice but it highlights the real issue
  
  shich's point about simulator mandates is the sharpest thing in this thread 
  
  esafak's cache economics point is underrated
I'm also pretty confident the @Marty McBot account they're replying to is also a bot but it's too new of account to say for sure:

  the .md scratch pad point is underrated, and the format matters more than people realize.
Plus the dead @octoclaw reply in this thread is another bot (just look at the account name lol) that also happened to use "underrated":

  The negative constraints thing is also underrated.
@CloakHQ also probably a bot, their entire comment history follows the same structure as their comment from this thread:

  The .md scratch pad between sessions is underrated

  The test harness point is the one that really sticks for me too
So far that's 3+ bot accounts I've seen so far in a single thread, the "Agentic" in the title/simonw as author may be a tempting target for people to throw their agents/claws at or it is just like catnip for them naturally.

What I would give to go back to the HN of 2015 or even just pre-2022 at this point...

Re: Agentic Engineering Patterns

#293

Post by Denzel »

ben30 wrote: post #60

I contribute to an open source spec based project management tool. I spend about a day back and forth iterating on a spec, using ai to refine the spec itself. Sometimes feeding it in and out of Claude/gemini telling each other where the feedback has come from. The spec is the value. Using the ai pm tool I break it down into n tasks and sub tasks and dependencies. I then trigger Claude in teams mode to accomplish the…

Mind linking the project so we can see the PR’s?

Re: Agentic Engineering Patterns

#294

Post by sn9 »

simonw wrote: post #180
jihadjihad wrote: post #171

Earlier quoted context omitted.

Agreed, and that's why I think adding some example prompts and ideas to the Testing section would be helpful. A vanilla-prompted LLM, in my experience, is very unreliable at adding tests that fail when the changes are reverted. Many times I've observed that the tests added by the model simply pass as part of the changes, but still pass even when those changes are no longer applied.

I had an example in that section but it got picked apart by pedants (who had good points) so I removed it. I plan to add another soon. You can still see it in the changelog: https://simonwillison.net/guides/agentic-engineering-pattern...

Matt Pocock has a nice TDD skill he's made available [0][1].

[0] https://www.aihero.dev/skill-test-driven-development-claude-...

[1] https://github.com/mattpocock/skills/blob/main/tdd/SKILL.md

Re: Agentic Engineering Patterns

#295

Post by lenocinor »

toraway wrote: post #292
sarkarsh wrote: post #216

Earlier quoted context omitted.

[dead]

BTW, check the comment history of the above account @sarkash, this is almost certainly an LLM replying with the exact same structure/format in all their comments. This is the underrated insight in the whole thread From comment history: This is good advice but it highlights the real issue shich's point about simulator mandates is the sharpest thing in this thread esafak's cache economics point is underrated I'm also p…

If you’re ok with it, I think emailing hn@ycombinator.com with this (which dang and the other mods read) would also be good.

Re: Agentic Engineering Patterns

#296

Post by epolanski »

slaye wrote: post #168

Simon, if you're reading this, I'd be really curious to hear your thoughts on how to effectively conduct code reviews in a world where "code is cheap". One of the biggest struggles I have on my team is coworkers straight up vibing parts of the code and not understanding or guiding the architecture of subsystems. Or at least, not writing code in a way that is meant to be understood by others. Then when I go through th…

Fire them. Easy.

They have to be responsible for what they push.

Re: Agentic Engineering Patterns

#297

Post by camgunz »

lmf4lol wrote: post #228
ep103 wrote: post #200

Earlier quoted context omitted.

Counter-point, developers that get used to not caring about function implementation, are going to culturally also not care as much about test implementation, making this proposed ideal impossible.

with LLMs, tests cost nearly nothing of effort but provide tremendous value.

> wow that's a lot of code, how will we ever review it?

>> have a model generate a bunch of tests instead

> wow that's a lot of test code, how will we know it's working correctly?

>> review it

> :face-with-rolling-eyes:

Re: Agentic Engineering Patterns

#298

Post by anukin »

simonw wrote: post #146
63stack wrote: post #140

Earlier quoted context omitted.

People are rushing to be the first one to coin something and hit it big. Imagine the amount of $$$ you could get for being an "expert ai consultant" in this space. There was already another attempt at agentic patterns earlier: https://agentic-patterns.com/ Absolute hot air garbage.

Which pieces of my writing are garbage?

I don’t think these kind of outbursts from some random guy in HN requires your response.

You have helped a lot of people from junior to staff+ level to understand how to use agents for software engineering using simple language. Calling it garbage is gross injustice to the work you put out.

Re: Agentic Engineering Patterns

#299

Post by atrevbot »

pkorzeniewski wrote: post #52
benrutter wrote: post #10

I use AI in my workflow mostly for simple boilerplate, or to troubleshoot issues/docs. I've dipped into agentic work now and again, but never been very impressed with the output (well, that there is any functioning output is insanely impressive, but it isn't code I want to be on the hook for complaining). I hear a lot of people saying the same, but similarly a bunch of people I respect saying they barely write code a…

One thing I rarely see mentioned is that often creating code by hand is simply faster (at least for me) than using AI. Creating a plan for AI, waiting for execution, verifying, prompting again etc. can take more time than just doing it on my own with a plan in my head (and maybe some notes). Creating something from scratch or doing advanced refactoring is almost always faster with AI, but most of my daily tasks are b…

I definitely agree with this and have experienced it as well. Having said that I wonder if the prevalence, and usefulness of AI will make those types of features fewer as intimate knowledge of the codebase decreases.

Re: Agentic Engineering Patterns

#300

Post by noddingham »

I'd choose a different word for the title of Hoard Things You Know How to Do. Hoarding is the opposite of what we want to do but I get from reading the section you mean create a collection that you can draw upon. IMO "Share" is a much better word choice.
Post reply on HN