Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents that run while I sleep

claudecodecamp.com

Post reply on HN
Print view View original

71–80 of 532 posts

Re: Agents that run while I sleep

#71

Post by SoftTalker »

magicalist wrote: post #47
egeozcan wrote: post #26

You can always tell claude to use red-green-refactor and that really is a step-up from "yeah don't forget to write tests and make sure they pass" at the end of the prompt, sure. But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. The trick is just not mixing/sharing the context. Differen…

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

A refactor should not affect the tests at all should it? If it does, it's more than a refactor.

Re: Agents that run while I sleep

#72

Post by Skidaddle »

egeozcan wrote: post #26

You can always tell claude to use red-green-refactor and that really is a step-up from "yeah don't forget to write tests and make sure they pass" at the end of the prompt, sure. But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. The trick is just not mixing/sharing the context. Differen…

How exactly do you set up your CC sessions to do this?

Re: Agents that run while I sleep

#73

Post by egeozcan »

skybrian wrote: post #66
egeozcan wrote: post #65

Earlier quoted context omitted.

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outco…

How do you define visibility rules? Is that possible for subagents?

AFAIK Claude doesn't support it, but if you're willing to go the extra mile, you can get creative with some bash script: https://pastebin.com/raw/m9YQ8MyS (generated this a second ago - just to get the point across )

To be clear, I don't do this. I never saw an agent cheat by peeking or something. I really did look through their logs.

I'd be very interested to see claude code and other tools support this pattern when dispatching agents to be really sure.

Re: Agents that run while I sleep

#74

Post by aray07 »

bhouston wrote: post #32

I call this "Test Theatre" and it is real. I wrote about it last year: https://benhouston3d.com/blog/the-rise-of-test-theater You have to actively work against it.

Test theatre is exactly the right framing. The tests are syntactically correct, they run, they pass but do they actually prove anything?

Re: Agents that run while I sleep

#75

Post by mattmanser »

InsideOutSanta wrote: post #63
tibbar wrote: post #21

Earlier quoted context omitted.

a lot of the value of tests is confirming that the system hasn't regressed beyond the behavior at the original release. It's bad if the original release is wrong, but a separate issue is if the system later accidentally stops behaving the way it did originally.

The issue I see is that the high test coverage created by having LLMs write tests results in almost all non-trivial changes breaking tests, even if they don't change behavior in ways that are visible from the outside. In one project I work, we require 100% test coverage, so people just have LLMs write tons of tests, and now every change I make to the code base always breaks tests. So now people just ignore broken tes…

I feel it end up a massive drag on development velocity and makes refactoring to simpler designs incredibly painful.

But hey, we're just supposed to let the AIs run wild and rewrite everything every change so maybe that's a heretic view.

Re: Agents that run while I sleep

#76

Post by SequoiaHope »

egeozcan wrote: post #26

You can always tell claude to use red-green-refactor and that really is a step-up from "yeah don't forget to write tests and make sure they pass" at the end of the prompt, sure. But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. The trick is just not mixing/sharing the context. Differen…

I’m telling it to use red/green tdd [1] and it will write test that don’t fail and then says “ah the issue is already fixed” and then move on. You really have to watch it very closely. I’m having a huge problem with bad tests in my system despite a “governance model” that I always refer it to which requires red/green tdd.

[1] https://simonwillison.net/guides/agentic-engineering-pattern...

Re: Agents that run while I sleep

#77

Post by magicalist »

SoftTalker wrote: post #71
magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

A refactor should not affect the tests at all should it? If it does, it's more than a refactor.

It depends on what you mean by "refactor" and how exactly you're testing, I guess, but that's not really at the heart of the point. red-green-refactor could also be used for adding new features, for instance, or an entire codebase, I guess.

Re: Agents that run while I sleep

#78

Post by gchamonlive »

SoftTalker wrote: post #71
magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

A refactor should not affect the tests at all should it? If it does, it's more than a refactor.

It can if your refactor needs to deal with interface changes, like moving methods around, changing argument order etc... all these need to propagate to the tests

Re: Agents that run while I sleep

#79

Post by vidimitrov »

He admits the real hole himself: "this doesn't catch spec misunderstandings. If your spec was wrong to begin with, the checks will pass."

But there's a second problem underneath that one. Acceptance criteria are ephemeral. You write them before prompting, Playwright runs against them, and then where do they go? A Notion doc. A PR comment. Nowhere permanent. Next time an agent touches that feature, it's starting from zero again.

The commit that ships the feature should carry the criteria that verified it. Git already travels with the code. The reasoning behind it should too.

Re: Agents that run while I sleep

#80

Post by gbnwl »

lagrange77 wrote: post #60
magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

> Reward hacking is very real and hard to guard against. Is it really about rewards? Im genuinely curious. Because its not a RL model.

I'm noticing terms related to DL/RL/NLP are being used more and more informally as AI takes over more of the cultural zeitgeist and people want to use the fancy new terms of the era, even if inaccurately. A friend told me he "trained and fine tuned a custom agent" for his work when what he meant was he modified a claude.md file.
Post reply on HN