Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backpressure is all you need

lucasfcosta.com

Post reply on HN
Print view View original

91–100 of 133 posts

Re: Backpressure is all you need

#91

Post by jaggederest »

osigurdson wrote: post #76
jaggederest wrote: post #71

Earlier quoted context omitted.

I don't mean just bugs, I mean any known issues. I test infra, I test UI, I test binary protocols, you name it. There is certainly no fast way to do it, even with AI (an AI generated suite is better than nothing but not as good), and it's a serious investment, but it's worth it. Testing becomes a process of correctness checking that snowballs over time, making everything else easier and better (or else the tests need…

Right. You mean all behaviors are tested, essentially. So if you / team are going to implement a new feature, what does that look like? Do you write Gherkin or similar, unit tests or both? Can you provide an example of what that might look like? How much of this has changed for you since the pre-AI days?

These days, yes, integration test at the high level (usually a 1-to-3 liner), then unit tests as I go, often some mocked functional tests. This is basically the same but a ton faster in the AI days, you have to hold the AI accountable and demand quality and iterate, but this weekend I've built an entire test suite for a monorepo I just started working on. It's garbage quality but better than no tests, of course, and will improve as I work.

You can find some open source examples on github, either directly https://github.com/pgdogdev/pgdog/commits/main/?author=jagge... or through my profile - that repo has a pure-sql integration suite I wrote essentially entirely with AI: https://github.com/pgdogdev/pgdog/tree/main/integration/sql

There's also older work on github you can see over the years, a mishmash and grab bag, I would prefer if more of my work were open source but somehow most employers still default to closed source

Edit: While I'm thinking about it, the other thing you can do with AI is demand that it TDD things - I'm more of a "test all the fucking time" adherent, I don't care whether the tests are written first, but AI is perfectly happy to skate by making a tautological test unless you make it write the test first, ensure it fails correctly, make your change, and don't let it modify the test.

Re: Backpressure is all you need

#92

Post by edfletcher_t137 »

_zoltan_ wrote: post #9

"In this post, I’ll cover a third, not-so-obvious approach: building ways for the agent to validate more of its own work before a human has to step in. " this has been an obvious thing to do since at least January (since Geoffrey Huntley published "everything is a ralph loop"), and this is how I've been working: build enough orchestration tooling to be able to automate everything: development container bringup, build…

yep, this has been obvious to a lot of people for awhile. especially after Cherny posted about exactly this in a massively-popular thread... four months ago: https://x.com/bcherny/status/2007179861115511237

Re: Backpressure is all you need

#93

Post by dodu_ »

_zoltan_ wrote: post #9

"In this post, I’ll cover a third, not-so-obvious approach: building ways for the agent to validate more of its own work before a human has to step in. " this has been an obvious thing to do since at least January (since Geoffrey Huntley published "everything is a ralph loop"), and this is how I've been working: build enough orchestration tooling to be able to automate everything: development container bringup, build…

> I understand this probably does not work if you're on some subscription and not using the API (tokens burn fast), but this has been extremely productive for me.

Do you have infinite money?

Re: Backpressure is all you need

#94

Post by hsaliak »

I have a custom agent that generates patches like you will with kernel development and I review and merge those in. https://github.com/hsaliak/std_slop/blob/main/docs/mail_mode...

My agent forces this workflow by disabling modifications outside the coding step.

I added looping to this not too long ago. https://github.com/hsaliak/std_slop/blob/main/docs/mail-loop...

This gives me the best of both worlds, hand curated reviews and automation. I often get the best quality if I do both, with an agent doing a pass first.

Re: Backpressure is all you need

#95

Post by ip26 »

lovich wrote: post #90
_zoltan_ wrote: post #9

"In this post, I’ll cover a third, not-so-obvious approach: building ways for the agent to validate more of its own work before a human has to step in. " this has been an obvious thing to do since at least January (since Geoffrey Huntley published "everything is a ralph loop"), and this is how I've been working: build enough orchestration tooling to be able to automate everything: development container bringup, build…

What’s the cost of all that though? I don’t doubt that productivity could be gained but when I see articles like the one on the Open Claw guy spending 1.3 million on tokens in a single month I am reminded of drag racing engines that can reach incredible speeds but also need to be completely rebuilt after a single race.

Depends on the quality of your validation loop. Can the agent find the bug in a five second unit test, or does it have to run the full deployment test?

It also presents tradeoffs in compute budget. Cycles spent executing large arrays of tests could mean less tokens spent debugging.

Re: Backpressure is all you need

#98

Post by erooke »

ovi256 wrote: post #83
atq2119 wrote: post #17

Earlier quoted context omitted.

That quote shows an utter disregard for basic human decency. It is the responsibility of the person running the coding agent to make sure the resulting PRs are high quality. Putting that on your team mates, or worse, random open source project maintainers on the internet, is the definition of an extractive contribution.

It seems the OP agrees with you, and he's proposing a method for how to do so using agents.

> It is the responsibility of the person running the coding agent to make sure the resulting PRs are high quality.

And

> he's proposing a method for how to do so using agents

Are not in agreement. The claim being made is that you shouldn't be sending PRs you haven't personally vetted to be high quality. Definitionally a bot cannot be used to personally vet something.

Re: Backpressure is all you need

#99

Post by jasonlotito »

I feel like a lot of people just forget you can put this stuff in pre-commit hooks. This forces the AI to deal with issues. You don't have to hope and pray it remembers your "Pretty please, check your work" markdown file.

A pre-commit hook has been wonderful. Sure, you can add instructions, but pre-commit hooks are where you want to put the guards.

Re: Backpressure is all you need

#100

Post by jasonlotito »

cadamsdotcom wrote: post #45

Everyone looking into this and other verification should be moving away from long prompts and complex skills, and looking into hooks. If you put all these checks in your stop hook and your git commit hook, your repo docs can tell your agent that checks will run automatically when it stops work, and it should fix any problems found. It’s wonderful to reintroduce determinism at the QA end of your process. I find it ver…

100% agree. None of this who watches the watchmen thing. Force it as a pre-commit hook. The best part there is it means you don't have to hope other people have setup there agents in the same way. It just works, every time.
Post reply on HN