Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents that run while I sleep

claudecodecamp.com

Post reply on HN
Print view View original

161–170 of 532 posts

Re: Agents that run while I sleep

#161

Post by 8note »

OsrsNeedsf2P wrote: post #42

Our app is a desktop integration and last year we added a local API that could be hit to read and interact with the UI. This unlocked the same thing the author is talking about - the LLM can do real QA - but it's an example of how it can be done even in non-web environments. Edit: I even have a skill called release-test that does manual QA for every bug we've ever had reported. It takes about 10 hours to run but I ex…

i got me a windows mcp setup running in a sandbox, so it can look at screenshots, see the UIA, and click things either by coordinate or by UIA.

i let it run overnight against a windows app i was working on, and that got it from mostly not working to mostly working.

the loop was

1. look at the code and specs to come up with tests 2. predict the result 3. try it 4. compare the prediction against rhe result 5. file bug report, or call it a success

and then switch to bug fixing, and go back around again. Worked really well in geminicli with the giant context window

Re: Agents that run while I sleep

#162

Post by mrbungie »

aprdm wrote: post #150
w4yai wrote: post #117

Earlier quoted context omitted.

You guys are describing wonderful things, but I've yet to see any implementation. I tried coding my own agents, yet the results were disappointing. What kind of setup do you use ? Can you share ? How much does it cost ?

If you are not spending 5-10k dollars a month for interesting projects, you likely won't see interesting results

I can't really tell if this is sarcasm or not.

Re: Agents that run while I sleep

#163

Post by joegaebel »

BeetleB wrote: post #2

I wish there was a way to "freeze" the tests. I want to write the tests first (or have Claude do it with my review), and then I want to get Claude to change the code to get them to pass - but with confidence that it doesn't edit any of the test files!

You'd be surprised - I know I was - you can encode Test-Driven development into workflows that agents actually follow. I wrote an in-depth guide about this and have a POC for people to try over here: https://www.joegaebel.com/articles/principled-agentic-softwa...

Re: Agents that run while I sleep

#164

Post by cheema33 »

dworks wrote: post #122
w4yai wrote: post #117

Earlier quoted context omitted.

You guys are describing wonderful things, but I've yet to see any implementation. I tried coding my own agents, yet the results were disappointing. What kind of setup do you use ? Can you share ? How much does it cost ?

rlm-workflow does all that TDD for you: https://skills.sh/doubleuuser/rlm-workflow/rlm-workflow (I built it)

Why make powershell a requirement? I like powershell, but Python is very common and already installed on many dev systems.

Re: Agents that run while I sleep

#165

Post by 8note »

storus wrote: post #50

Wasn't the best practice to run one model/coding agent that writes the code and another one that reviews it? E.g. Claude Code for writing the code, GPT Codex to review/critique it? Different reward functions.

even in one agent, a different starting prompt will have you tracing a very different path through the model.

maybe it still sends you to the same valley, but there's so many parameters and dimensions that i dont think its very likely without also being correct

Re: Agents that run while I sleep

#166

Post by skybrian »

ssk42 wrote: post #127
skybrian wrote: post #84

Earlier quoted context omitted.

So what do you do? Do you define roles somewhere and tell the agent to assign these roles to subagents?

Fun to see you not on tildes. Setting up a clean room is one of the only ways to do Evals on agentic harnesses. Especially prevalent with Windsurf which doesn’t have an easy CLI start. So how? The easiest answer when allowed is docker. Literally new image per prompt. There’s also flags with Claude to not use memory and from there you can use -p to have it just be like a normal cli tool. Windsurf requires manual effor…

Sounds interesting, but I'm not quite getting the relevance for people writing code with an agent. Should I be doing evals?

Re: Agents that run while I sleep

#167

Post by hinkley »

InsideOutSanta wrote: post #63
tibbar wrote: post #21

Earlier quoted context omitted.

a lot of the value of tests is confirming that the system hasn't regressed beyond the behavior at the original release. It's bad if the original release is wrong, but a separate issue is if the system later accidentally stops behaving the way it did originally.

The issue I see is that the high test coverage created by having LLMs write tests results in almost all non-trivial changes breaking tests, even if they don't change behavior in ways that are visible from the outside. In one project I work, we require 100% test coverage, so people just have LLMs write tons of tests, and now every change I make to the code base always breaks tests. So now people just ignore broken tes…

My best unit tests are 3 lines, one of them whitespace, and they assert one single thing that's in the requirements.

These are the only tests I've witnessed people delete outright when the requirements change. Anything more complex than this, they'll worry that there's some secondary assertion being implied by a test so they can't just delete it.

Which, really is just experience telling them that the code smells they see in the tests are actually part of the test.

meanwhile:

    it("only has one shipping address", ...
is demonstrably a dead test when the story is, "allow users to have multiple shipping addresses", as is a test that makes sure balances can't go negative when we decide to allow a 5 day grace period on account balances. But if it's just one of six asserts in the same massive tests, then people get nervous and start losing time.

Re: Agents that run while I sleep

#168

Post by hinkley »

ForHackernews wrote: post #96
InsideOutSanta wrote: post #63

Earlier quoted context omitted.

The issue I see is that the high test coverage created by having LLMs write tests results in almost all non-trivial changes breaking tests, even if they don't change behavior in ways that are visible from the outside. In one project I work, we require 100% test coverage, so people just have LLMs write tons of tests, and now every change I make to the code base always breaks tests. So now people just ignore broken tes…

Unit tests vs acceptance tests. You shouldn't be afraid to throw away unit tests if the implementation changes, and acceptance tests should verify behavior at API boundaries, ignoring implementation details.

BDD helps with this as it can allow you to get the setup out of the tests making it even cheaper for someone to yeet a defunct test.

Re: Agents that run while I sleep

#169

Post by hinkley »

jdlshore wrote: post #27

Pet peeve: this post misunderstands “TDD.” What it really describes is acceptance tests. TDD is a tool for working in small steps, so you get continuous feedback on your work as you go, and so you can refine your design based on how easy it is to use in practice. It’s “red green refactor repeat”, and each step is only a handful of lines of code. TDD is not “write the tests, then write the code.” It’s “write the tests…

suggestion: TeDD talk.

Re: Agents that run while I sleep

#170

Post by joegaebel »

fragmede wrote: post #24

Adversarial AI code gen. Have another AI write the tests, tell Codex that Claude wrote some code and to audit the code and write some tests. Tell Gemini that Codex wrote the tests. Have it audit the tests. Tell Codex that Gemini thinks its code is bad and to do better. (Have Gemini write out why into dobetter.md)

Even better, encode it into a workflow and have the subagents be adversarial to each other: https://www.joegaebel.com/articles/principled-agentic-softwa...
Post reply on HN