Agents that run while I sleep • Hacker News Board

Re: Agents that run while I sleep

#111

Post by itissid » Tue, Mar 10, 2026, 10:23 PM UTC

Many times there is really no way of getting around some of the expert-human judgement complexity of the larger question of "How to get agents to build reliably".

One example I have been experimenting is using Learning Tests[1]. The idea is that when something new is introduced in the system the Agent must execute a high value test to teach itself how to use this piece of code. Because these should be high leverage i.e. they can really help any one understand the code base better, they should be exceptionally well chosen for AIs to use to iterate. But again this is just the expert-human judgement complexity shifted to identifying these for AI to learn from. In code bases that code Millions of LoC in new features in days, this would require careful work by the human.

[1] https://anthonysciamanna.com/2019/08/22/the-continuous-value...

Re: Agents that run while I sleep

#112

Post by xandrius » Tue, Mar 10, 2026, 10:24 PM UTC

paganel wrote: post #98

genghisjahn wrote: post #85

Earlier quoted context omitted.

I went the same way. At first I was splitting off work trees and running all the agents that I could afford, then I realized I just can't keep up with it all, running few agents around one issue in one directory is fast enough. Way faster than before and I can still follow what's happening.

> off work trees and running all the agents that I could afford, I still think that we, programmers, having to pay money in order to write code is a travesti. And I'm not talking about paying the license for the odd text editor or even for an operating system, I'm talking about day-to-day operations. I'm surprised that there isn't a bigger push-back against this idea.

It's silly, who wouldn't answer yes to the question "would you like to finish your task faster?". The real trick is to produce more but by putting less effort than before.

Re: Agents that run while I sleep

#113

Post by xandrius » Tue, Mar 10, 2026, 10:25 PM UTC

storus wrote: post #50

Wasn't the best practice to run one model/coding agent that writes the code and another one that reviews it? E.g. Claude Code for writing the code, GPT Codex to review/critique it? Different reward functions.

I think people are misunderstanding reward functions and LLMs.

LLMs don't actually have a reward system like some other ML models.

Re: Agents that run while I sleep

#114

Post by xandrius » Tue, Mar 10, 2026, 10:27 PM UTC

lateforwork wrote: post #19

> When Claude writes tests for code Claude just wrote, it's checking its own work. You can have Gemini write the tests and Claude write the code. And have Gemini do review of Claude's implementation as well. I routinely have ChatGPT, Claude and Gemini review each other's code. And having AI write unit tests has not been a problem in my experience.

I don't think that's necessary, just make sure the context is not shared. A pretty good model can handle both sides well enough.

Re: Agents that run while I sleep

#115

Post by xienze » Tue, Mar 10, 2026, 10:30 PM UTC

egeozcan wrote: post #65

magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outco…

This seems like a tremendous amount of planning, babysitting, verification, and token cost just to avoid writing code and tests yourself.

Re: Agents that run while I sleep

#116

Post by habinero » Tue, Mar 10, 2026, 10:34 PM UTC

xienze wrote: post #115

egeozcan wrote: post #65

Earlier quoted context omitted.

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outco…

This seems like a tremendous amount of planning, babysitting, verification, and token cost just to avoid writing code and tests yourself.

It's assigning yourself the literal worst parts of the job - writing specs, docs, tests and reading someone else's code.

Re: Agents that run while I sleep

#117

Post by w4yai » Tue, Mar 10, 2026, 10:35 PM UTC

egeozcan wrote: post #65

magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outco…

You guys are describing wonderful things, but I've yet to see any implementation. I tried coding my own agents, yet the results were disappointing.

What kind of setup do you use ? Can you share ? How much does it cost ?

Re: Agents that run while I sleep

#118

Post by apsdsm » Tue, Mar 10, 2026, 10:39 PM UTC

Do you really, honestly, have to be doing this stuff even when you sleep? To the point it hits you “wait is this even any good? Gee I don’t want to push out slop.”

If you don’t trust the agent to do it right in the first place why do you trust them to implement your tests properly? Nothing but turtles here.

Re: Agents that run while I sleep

#119

Post by fwip » Tue, Mar 10, 2026, 10:44 PM UTC

jeremyjh wrote: post #110

paganel wrote: post #98

Earlier quoted context omitted.

> off work trees and running all the agents that I could afford, I still think that we, programmers, having to pay money in order to write code is a travesti. And I'm not talking about paying the license for the odd text editor or even for an operating system, I'm talking about day-to-day operations. I'm surprised that there isn't a bigger push-back against this idea.

What is strange about paying for tools that improve productivity? Unless you consider your own time worthless you should always be open to spending more to gain more.

Are the jobs out there actually paying people more?

Re: Agents that run while I sleep

#120

Post by what » Tue, Mar 10, 2026, 10:45 PM UTC

jeremyjh wrote: post #110

paganel wrote: post #98

Earlier quoted context omitted.

> off work trees and running all the agents that I could afford, I still think that we, programmers, having to pay money in order to write code is a travesti. And I'm not talking about paying the license for the odd text editor or even for an operating system, I'm talking about day-to-day operations. I'm surprised that there isn't a bigger push-back against this idea.

What is strange about paying for tools that improve productivity? Unless you consider your own time worthless you should always be open to spending more to gain more.

Your own time is worthless if you’re not spending it doing something that makes more money. You don’t make more money increasing your productivity for work when you’re expected to work the same number of hours.