Agents that run while I sleep • Hacker News Board

Re: Agents that run while I sleep

#101

Post by gonzalohm » Tue, Mar 10, 2026, 10:10 PM UTC

SoftTalker wrote: post #18

RealityVoid wrote: post #4

It's... really the same problem when you hire people to just write tests. A lot of time it just confirms that the code does what the code does. Having clear specs of what the code should do make things better and clearer.

Yep, tests written after the fact are just verifying tautologies. > Most teams don't [write tests first] because thinking through what the code should do before writing it takes time they don't have. It's astonishing to me how much our industry repeats the same mistakes over and over. This doesn't seem like what other engineering disciplines do. Or is this just me not knowing what it looks like behind the curtain of…

That's probably dependent on your specific area of work. For most projects, It's okayish to deploy code with bugs. There will be future releases that fix those bugs and add improvements. Obviously that's not the case with high risk systems like space rockets software and similar.

With other engineering professions, all projects are like that. You cannot "deploy a bridge to production" to see what happens and fix it after a few have died

Re: Agents that run while I sleep

#102

Post by [deleted] » Tue, Mar 10, 2026, 10:10 PM UTC

RealityVoid wrote: post #4

It's... really the same problem when you hire people to just write tests. A lot of time it just confirms that the code does what the code does. Having clear specs of what the code should do make things better and clearer.

[deleted]

Re: Agents that run while I sleep

#103

Post by JBorrow » Tue, Mar 10, 2026, 10:11 PM UTC

bhouston wrote: post #32

I call this "Test Theatre" and it is real. I wrote about it last year: https://benhouston3d.com/blog/the-rise-of-test-theater You have to actively work against it.

Yeah, having your agent write 3x the code in exhaustive tests (I tried this recently and got 600 lines of tests for my 100 lines of code!) sure makes things look great, but when you actually look at the content of the tests they’re meaningless. Good tests validate the use of design patterns, ensure that dependencies hold, and are meaningful (e.g. shortcut debugging by setting up useful state) when they break.

Re: Agents that run while I sleep

#104

Post by emirhan_demir » Tue, Mar 10, 2026, 10:13 PM UTC

A short story: A developer let ClaudeCode manage his AWS infrastructure. The agent ran a Terraform destroy command... Gone: 2 websites, production database, all backups and 2.5 years of data The agent didn't make a mistake. It did exactly what it was allowed to do. That's the problem dude

Re: Agents that run while I sleep

#105

Post by rglover » Tue, Mar 10, 2026, 10:16 PM UTC

daxfohl wrote: post #91

Sounds like we've just gotten into lazy mode where we believe that whatever it spits out is good enough. Or rather, we want to believe it, and convince ourselves that some simple guardrail we put up will make it true, because God forbid we have to use our own brain again. What if instead, the goal of using agents was to increase quality while retaining velocity, rather than the current goal of increasing velocity whi…

You can't. To retain and improve quality requires care. Very few if any of the people setting stuff like this up truly care about delivering a quality result (any result is the real goal). Unless there's some incentive to care, quality will be found among the exceedingly rare people/businesses.

Re: Agents that run while I sleep

#106

Post by osigurdson » Tue, Mar 10, 2026, 10:18 PM UTC

egeozcan wrote: post #26

You can always tell claude to use red-green-refactor and that really is a step-up from "yeah don't forget to write tests and make sure they pass" at the end of the prompt, sure. But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. The trick is just not mixing/sharing the context. Differen…

So more stuff happens with this approach but how do you know what it generates is correct?

Re: Agents that run while I sleep

#107

Post by overfeed » Tue, Mar 10, 2026, 10:19 PM UTC

> At some point you're not reviewing diffs at all, just watching deploys and hoping something doesn't break.

To everyone who plan on automating themselves out of a job by taking the human element out- this is the endgame that management wants: replacing your (expensive and non-tax-optimized) labor with scalable Opex.

Re: Agents that run while I sleep

#108

Post by osigurdson » Tue, Mar 10, 2026, 10:21 PM UTC

I think the solution has to be end to end tests. Maybe first run by humans, then maybe agents can learn and replicate. I can't see why unit tests really help other than for the LLM to reason about its own code a little more.

Re: Agents that run while I sleep

#109

Post by seanmcdirmid » Tue, Mar 10, 2026, 10:22 PM UTC

jeremyjh wrote: post #88

seanmcdirmid wrote: post #44

I've been doing differential testing in Gemini CLI using sub-agents. The idea is: 1. one agent writes/updates code from the spec 2. one agent writes/updates tests from identified edge cases in the spec. 3. a QA agent runs the tests against the code. When a test fails, it examines the code and the test (the only agent that can see both) to determine blame, then gives feedback to the code and/or test writing agent on w…

Where is the interface defined ? If it is just the coder reading the test it can hard code specific cases based on the test setup/fixture data.

There is a specification and the interface is defined from that. The coder never gets to see the test.

Re: Agents that run while I sleep

#110

Post by jeremyjh » Tue, Mar 10, 2026, 10:23 PM UTC

paganel wrote: post #98

genghisjahn wrote: post #85

Earlier quoted context omitted.

I went the same way. At first I was splitting off work trees and running all the agents that I could afford, then I realized I just can't keep up with it all, running few agents around one issue in one directory is fast enough. Way faster than before and I can still follow what's happening.

> off work trees and running all the agents that I could afford, I still think that we, programmers, having to pay money in order to write code is a travesti. And I'm not talking about paying the license for the odd text editor or even for an operating system, I'm talking about day-to-day operations. I'm surprised that there isn't a bigger push-back against this idea.

What is strange about paying for tools that improve productivity? Unless you consider your own time worthless you should always be open to spending more to gain more.