Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Agents that run while I sleep

claudecodecamp.com

Post reply on HN
Print view View original

221–230 of 532 posts

Re: Agents that run while I sleep

#221

Post by interpol_p »

The example given in the article is acceptance criteria for a login/password entry flow. This is fairly easy to spec-out in terms of AC and TDD.

I have been asking these tools to build other types of projects where it (seems?) much more difficult to verify without a human-in-the-loop. One example is I had asked Codex to build a simulation of the solar system using a Metal renderer. It produced a fun working app quickly.

I asked it to add bloom. It looped for hours, failing. I would have to manually verify — because even from images — it couldn't tell what was right and wrong. It only got it right when I pasted a how-to-write-a-bloom-shader-pass-in-Metal blog post into it.

Then I noticed that all of the planet textures were rotating oddly every time I orbited the camera. Codex got stuck in another endless loop of "Oh, the lookAt matrix is in column major, let me fix that ." or focusing (incorrectly) on UV coordinates and shader code. Eventually Codex told me what I was seeing "was expected" and that I just "felt like it was wrong."

When I finally realised the problem was that Codex had drawn the planets with back-facing polygons only, I reported the error, to which Codex replied, "Good hypothesis, but no"

I insisted that it change the culling configuration and then it worked fine.

These tools are fun, and great time savers (at times), but take them out of their comfort zone and it becomes real hard to steer them without domain knowledge and close human review.

Re: Agents that run while I sleep

#222

Post by ziofill »

> Writing acceptance criteria is harder than writing a prompt, because it forces you to think through edge cases before you've seen them. Engineers resist it for the same reason they resisted TDD, because it feels slower at the start.

This resonates with my experience, and it is also a refreshing honest take: pushing back on heavy upfront process isn't laziness, it's just the natural engineers drive to build things and feel productive.

Re: Agents that run while I sleep

#223

Post by Gigachad »

mewpmewp2 wrote: post #209
jack_pp wrote: post #208

Earlier quoted context omitted.

what in God's Name could you do in PHP that you can't do in a modern framework?

You can build those things in modern frameworks, it will just be more headache and will feel outdated in 6 months.

6 years ago I was writing apps in typescript and react, if I was starting a new project today I'd write it in typescript and react.

Re: Agents that run while I sleep

#224

Post by collingreen »

gbnwl wrote: post #80
lagrange77 wrote: post #60

Earlier quoted context omitted.

> Reward hacking is very real and hard to guard against. Is it really about rewards? Im genuinely curious. Because its not a RL model.

I'm noticing terms related to DL/RL/NLP are being used more and more informally as AI takes over more of the cultural zeitgeist and people want to use the fancy new terms of the era, even if inaccurately. A friend told me he "trained and fine tuned a custom agent" for his work when what he meant was he modified a claude.md file.

Respectfully, your friend doesn't know what he is talking about and is saying things that just "feel right" (vibe talking??). Which might be exactly how technical terms lose their meaning so perhaps you're exactly right.

Re: Agents that run while I sleep

#225

Post by esperent »

egeozcan wrote: post #65
magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outco…

Someone directly down from you suggested looking up Mike Postock's TDD skill, so I did:

https://github.com/mattpocock/skills/blob/main/tdd%2FSKILL.m...

Everything below quoted from that skill, and serves as a much better rebuttal than I had started writing:

DO NOT write all tests first, then all implementation. This is "horizontal slicing" - treating RED as "write all tests" and GREEN as "write all code."

This produces crap tests:

Tests written in bulk test imagined behavior, not actual behavior You end up testing the shape of things (data structures, function signatures) rather than user-facing behavior Tests become insensitive to real changes - they pass when behavior breaks, fail when behavior is fine

You outrun your headlights, committing to test structure before understanding the implementation

Correct approach:

Vertical slices via tracer bullets.

One test → one implementation → repeat. Each test responds to what you learned from the previous cycle. Because you just wrote the code, you know exactly what behavior matters and how to verify it.

Re: Agents that run while I sleep

#226

Post by lmz »

tdeck wrote: post #216
xandrius wrote: post #112

Earlier quoted context omitted.

It's silly, who wouldn't answer yes to the question "would you like to finish your task faster?". The real trick is to produce more but by putting less effort than before.

> who wouldn't answer yes to the question "would you like to finish your task faster?" People who enjoy the process of completing the task?

Maybe we'd see "coding gyms" like how white collar workers have gyms for the physical exercise they're not getting from their work.

Re: Agents that run while I sleep

#227

Post by JumpCrisscross »

timr wrote: post #211
JumpCrisscross wrote: post #196

Earlier quoted context omitted.

> Am I supposed to be impressed by this? No. But it is noteworthy. A lot of what one previously needed a SWE to do can now be brute forced well enough with AI. (Granted, everything SWEs complained about being tedious.) From the customer’s perspective, waiting for buggy code tomorrow from San Francisco, buggy code tonight from India or buggy code from an AI at 4AM aren’t super different for maybe two thirds of use cas…

> A lot of what one previously needed a SWE to do can now be brute forced well enough with AI. (Granted, everything SWEs complained about being tedious.) Only if you ignore everything they generate. Look at all the comments saying that the agent hallucinates a result, generates always-passing tests, etc. Those are absolutely true observations -- and don't touch on the fact that tests can pass, the red/green approach…

> coders think that these random text generators are trustworthy enough to run unsupervised, we've moved all of this chaotic energy up a level

But it works well enough for most use cases. Most of what we do isn’t life or death.

Re: Agents that run while I sleep

#228

Post by seer »

egeozcan wrote: post #65
magicalist wrote: post #47

Earlier quoted context omitted.

> But even better, tell it to create subagents to form red team, green team and refactor team while the main instance coordinates them, respecting the clean-room rules. It really works. It helps, but it definitely doesn't always work, particularly as refactors go on and tests have to change. Useless tests start grow in count and important new things aren't tested or aren't tested well. I've had both Opus 4.6 and Code…

The trick is, with the setup I mentioned, you change the rewards. The concept is: Red Team (Test Writers), write tests without seeing implementation. They define what the code should do based on specs/requirements only. Rewarded by test failures. A new test that passes immediately is suspicious as it means either the implementation already covers it (diminishing returns) or the test is tautological. Red's ideal outco…

This seems quite amazing really, thanks for sharing

What is the scope of projects / features you’ve seen this be successful at?

Do you have a step before where an agent verifies that your new feature spec is not contradictory, ambiguous etc. Maybe as reviewed with regards to all the current feature sets?

Do you make this a cycle per step - by breaking down the feature to small implementable and verifiable sub-features and coding them in sequence, or do you tell it to write all the tests first and then have at it with implementation and refactoring?

Why not refactor-red-green-refactor cycle? E.g. a lot of the time it is worth refactoring the existing code first, to make a new implementation easier, is it worth encoding this into the harness?

Re: Agents that run while I sleep

#229

Post by firstdata »

The hardest part of running agents autonomously is the data quality problem. When your agent runs unsupervised, every decision is only as good as the data it pulls. Having agents access authoritative structured sources (government APIs, international org datasets) rather than scraping random pages makes a huge difference. The real failure mode is not hallucination - it is the agent confidently acting on unreliable data.

Re: Agents that run while I sleep

#230

Post by ehnto »

Gigachad wrote: post #223
mewpmewp2 wrote: post #209

Earlier quoted context omitted.

You can build those things in modern frameworks, it will just be more headache and will feel outdated in 6 months.

6 years ago I was writing apps in typescript and react, if I was starting a new project today I'd write it in typescript and react.

People bicker about PHP and Javascript, sorry Typescript, like they aren't both mule languages peoppe pick up to get work done. They both matured really well through years of production use.

They are in the same group, similar pedigree. If you were programming purely for the art of it, you would have had time to discover much nicer languages than either, but that's not what most people are doing so it doesn't really matter. They're different but they're about as good as eachother.

Post reply on HN