Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents that run while I sleep

claudecodecamp.com

Post reply on HN
Print view View original

91–100 of 532 posts

Re: Agents that run while I sleep

#91

Post by daxfohl »

Sounds like we've just gotten into lazy mode where we believe that whatever it spits out is good enough. Or rather, we want to believe it, and convince ourselves that some simple guardrail we put up will make it true, because God forbid we have to use our own brain again.

What if instead, the goal of using agents was to increase quality while retaining velocity, rather than the current goal of increasing velocity while (trying to) retain quality? How can we make that world come to be? Because TBH that's the only agentic-oriented future that seems unlikely to end in disaster.

Re: Agents that run while I sleep

#92

Post by hexaga »

lagrange77 wrote: post #60
magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

> Reward hacking is very real and hard to guard against. Is it really about rewards? Im genuinely curious. Because its not a RL model.

There is a nontrivial amount of RL training (RLHF, RLVR, ...), so it would be reasonable to call it an RL model.

And with that comes reward hacking - which isn't really about looking for more reward but rather that the model has learned patterns of behavior that got reward in the train env.

That is, any kind of vulnerability in the train env manifests as something you'd recognize as reward hacking in the real world: making tests pass _no matter what_ (because the train env rewarded that behavior), being wildly sycophantic (because the human evaluators rewarded that behavior), etc.

Re: Agents that run while I sleep

#94

Post by dwaltrip »

vidimitrov wrote: post #79

He admits the real hole himself: "this doesn't catch spec misunderstandings. If your spec was wrong to begin with, the checks will pass." But there's a second problem underneath that one. Acceptance criteria are ephemeral. You write them before prompting, Playwright runs against them, and then where do they go? A Notion doc. A PR comment. Nowhere permanent. Next time an agent touches that feature, it's starting from…

Did AI write this?

Re: Agents that run while I sleep

#95

Post by ge96 »

recroad wrote: post #83

Am I supposed to be impressed by this? I think people are now just using agents for the sake of it. I'm perfectly happy running two simple agents, one for writing and one for reviewing. I don't need to go be writing code at faster than light speed. Just focusing on the spec, and watching the agent as it does its work and intervening when it goes sideways is perfectly fine with me. I'm doing 5-7x productivity easily,…

I would be impressed if I could say "here's $100 turn it into $1000" but you still gotta do the thinking.

Re: Agents that run while I sleep

#96

Post by ForHackernews »

InsideOutSanta wrote: post #63
tibbar wrote: post #21

Earlier quoted context omitted.

a lot of the value of tests is confirming that the system hasn't regressed beyond the behavior at the original release. It's bad if the original release is wrong, but a separate issue is if the system later accidentally stops behaving the way it did originally.

The issue I see is that the high test coverage created by having LLMs write tests results in almost all non-trivial changes breaking tests, even if they don't change behavior in ways that are visible from the outside. In one project I work, we require 100% test coverage, so people just have LLMs write tons of tests, and now every change I make to the code base always breaks tests. So now people just ignore broken tes…

Unit tests vs acceptance tests. You shouldn't be afraid to throw away unit tests if the implementation changes, and acceptance tests should verify behavior at API boundaries, ignoring implementation details.

Re: Agents that run while I sleep

#97

Post by bluGill »

gchamonlive wrote: post #78
SoftTalker wrote: post #71

Earlier quoted context omitted.

A refactor should not affect the tests at all should it? If it does, it's more than a refactor.

It can if your refactor needs to deal with interface changes, like moving methods around, changing argument order etc... all these need to propagate to the tests

Your tests are an assertion that 'no matter what this will never change'. If your interface can change then you are testing implementation details instead of the behavior users care about.

the above is really hard. A lot of tdd 'experts' don't understand is and teach fragile tests that are not worth having.

Re: Agents that run while I sleep

#98

Post by paganel »

genghisjahn wrote: post #85
recroad wrote: post #83

Am I supposed to be impressed by this? I think people are now just using agents for the sake of it. I'm perfectly happy running two simple agents, one for writing and one for reviewing. I don't need to go be writing code at faster than light speed. Just focusing on the spec, and watching the agent as it does its work and intervening when it goes sideways is perfectly fine with me. I'm doing 5-7x productivity easily,…

I went the same way. At first I was splitting off work trees and running all the agents that I could afford, then I realized I just can't keep up with it all, running few agents around one issue in one directory is fast enough. Way faster than before and I can still follow what's happening.

> off work trees and running all the agents that I could afford,

I still think that we, programmers, having to pay money in order to write code is a travesti. And I'm not talking about paying the license for the odd text editor or even for an operating system, I'm talking about day-to-day operations. I'm surprised that there isn't a bigger push-back against this idea.

Re: Agents that run while I sleep

#99

Post by vidimitrov »

dwaltrip wrote: post #94
vidimitrov wrote: post #79

He admits the real hole himself: "this doesn't catch spec misunderstandings. If your spec was wrong to begin with, the checks will pass." But there's a second problem underneath that one. Acceptance criteria are ephemeral. You write them before prompting, Playwright runs against them, and then where do they go? A Notion doc. A PR comment. Nowhere permanent. Next time an agent touches that feature, it's starting from…

Did AI write this?

Nope - though I’ll take it as a compliment either way. It’s a problem I’ve been sitting with for a while, so the answer came out more formed than I expected. You disagree?

Re: Agents that run while I sleep

#100

Post by achierius »

egeozcan wrote: post #73
skybrian wrote: post #66

Earlier quoted context omitted.

How do you define visibility rules? Is that possible for subagents?

AFAIK Claude doesn't support it, but if you're willing to go the extra mile, you can get creative with some bash script: https://pastebin.com/raw/m9YQ8MyS (generated this a second ago - just to get the point across ) To be clear, I don't do this. I never saw an agent cheat by peeking or something. I really did look through their logs. I'd be very interested to see claude code and other tools support this pattern when…

> To be clear, I don't do this.

How do you know that it works then? Are you using a different tool that does support it?

Post reply on HN