Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backpressure is all you need

lucasfcosta.com

Post reply on HN
Print view View original

61–70 of 133 posts

Re: Backpressure is all you need

#61

Post by osigurdson »

_zoltan_ wrote: post #60
osigurdson wrote: post #54

Earlier quoted context omitted.

I can see how you could avoid regressions this way, but what do you add to your harness to prove that a new feature is working?

for us it's (usually) very easy as I work on performance optimization. a non-negligible part of this is correctness and verifiability, so we already have some of that. to give you an example just recently I've coded a feature that for our shuffle operation can report which channel did the bytes flow through (as the PR giving us the plumbing underneath has landed upstream recently). what this basically means is that y…

Definitely agree that performance optimization is a good use case for LLMs. Here you have both a measurable goal / objective function and guardrails against functional regressions. It kind of closes the loop in that regard.

One thing however is a test suite is not usually exhaustive in the sense that any code that passes the tests is valid. Usually tests are more complimentary in nature. Therefore you could still possibly get code degradation, potentially.

Re: Backpressure is all you need

#62

Post by _zoltan_ »

dirtbag__dad wrote: post #53
_zoltan_ wrote: post #9

"In this post, I’ll cover a third, not-so-obvious approach: building ways for the agent to validate more of its own work before a human has to step in. " this has been an obvious thing to do since at least January (since Geoffrey Huntley published "everything is a ralph loop"), and this is how I've been working: build enough orchestration tooling to be able to automate everything: development container bringup, build…

You can get really far with the 20x Claude Code and Codex plans. They are many orders of magnitude cheaper than api calls.

Agreed! Until they fit. :-)

Re: Backpressure is all you need

#63

Post by jaggederest »

osigurdson wrote: post #61
_zoltan_ wrote: post #60

Earlier quoted context omitted.

for us it's (usually) very easy as I work on performance optimization. a non-negligible part of this is correctness and verifiability, so we already have some of that. to give you an example just recently I've coded a feature that for our shuffle operation can report which channel did the bytes flow through (as the PR giving us the plumbing underneath has landed upstream recently). what this basically means is that y…

Definitely agree that performance optimization is a good use case for LLMs. Here you have both a measurable goal / objective function and guardrails against functional regressions. It kind of closes the loop in that regard. One thing however is a test suite is not usually exhaustive in the sense that any code that passes the tests is valid. Usually tests are more complimentary in nature. Therefore you could still pos…

> One thing however is a test suite is not usually exhaustive in the sense that any code that passes the tests is valid. Usually tests are more complimentary in nature.

Not in the world of AI - if your tests don't catch any known issues, the problem is the tests aren't comprehensive enough. There's no excuse at this point not to have an incredibly comprehensive test suite, to go with your other agent feedback loop constraints

Re: Backpressure is all you need

#64

Post by NamTaf »

dcrazy wrote: post #58
NamTaf wrote: post #28

Earlier quoted context omitted.

Never mind stumbling into proper engineering principles like having documented, testable requirements specifications.

I”ve been pretty happy with this side effect of the agentic coding bubble.

As a non-tech engineer (mechanical, trains) it's fascinating seeing what is essentially the "not real engineers" SWE crew finally pay the piper because they've invoked what is in essence a non-compliant, cost-focused subcontractor and now need all of the same engineering rigours they never previously understood.

Re: Backpressure is all you need

#65

Post by monkpit »

pshirshov wrote: post #35

A very long post about a simple and very obvious idea with many different implementations. The three main problems are 1) API usage is deadly expensive 2) Claude is about to make all automation very expensive 3) all the flows where a model has the initiative are strictly biased towards unwarranted stops (checkpointing). Also, I won't call that "backpressure", there is no producer-consumer disbalance or something simi…

> 2) Claude is about to make all automation very expensive

Wait, what happened here??

Re: Backpressure is all you need

#66

Post by bluGill »

0x696C6961 wrote: post #59
bluGill wrote: post #47

Earlier quoted context omitted.

Every large project in the coming back to waterfall. While the problems are certainly known and it was ultimately developed as a straw man, everything else ends up working worse. That said, you shouldn't be thinking pure waterfall as it's drawn up as a strawman, but rather a waterfall variation with feedback loops. But in the end, in very, very many cases, you have to know an end date in order to get things done beca…

"waterfall variation with feedback loops" lol next we're going to have "agile where you plan everything up front"

Pretty much what most agile ends up. Plans are worthless but planning is a valuable exercise. (Attributed to various generals)

Re: Backpressure is all you need

#67

Post by kami23 »

pstorm wrote: post #57
kami23 wrote: post #56

Earlier quoted context omitted.

I have it record a series of gifs or videos that I look over. If something looks off I'll dig into it, but I break down work into very very small chunks that are usually easily verifiable or don't require multiple steps. Another thing I have in the general sdlc process is having it add enough logging to verify features are turned on, configured as we expected, and that becomes enough feedback for most of my features.…

I’ve been building this out too, and your comment made me realize the missing piece for me. I’ve given the agents tools to validate its own work, but I haven’t improved the experience of humans verifying the agents’ work.

For video/image stuff I found the ability for the LLMs to use ffmpeg and imagemagick to be quite fun.

Re: Backpressure is all you need

#68

Post by jarrettcoggin »

jon-wood wrote: post #43

This what hooks[1] are for, except hooks allow specifying criteria in certain conditions (like the agent believing it’s done and ready to hand back to the user) in a manner that the agent won’t just forget about once it’s a few turns deep, and doesn’t require triggering a whole other LLM instance to read some plain text instructions while you hope it interprets them correctly. It absolutely makes sense to have a syst…

I was thinking the exact same thing. There are multiple places to implement hooks (git hooks, Claude hooks, etc.).

One thing I've been wondering about is how to reliably protect specific portions of the system from unexpected/unnecessary change (for example, a failing test that Claude decides to comment out or rewrite to get it to pass). My only thought for this was to automatically revert test changes during specific portions of the implementation, but that feels overly rigid and potentially prevents things like refactoring code.

Re: Backpressure is all you need

#69

Post by pshirshov »

monkpit wrote: post #65
pshirshov wrote: post #35

A very long post about a simple and very obvious idea with many different implementations. The three main problems are 1) API usage is deadly expensive 2) Claude is about to make all automation very expensive 3) all the flows where a model has the initiative are strictly biased towards unwarranted stops (checkpointing). Also, I won't call that "backpressure", there is no producer-consumer disbalance or something simi…

> 2) Claude is about to make all automation very expensive Wait, what happened here??

They will charge use of -p and agent SDK at API rates since 14 June. So, x20..x50 price increase.

Re: Backpressure is all you need

#70

Post by osigurdson »

jaggederest wrote: post #63
osigurdson wrote: post #61

Earlier quoted context omitted.

Definitely agree that performance optimization is a good use case for LLMs. Here you have both a measurable goal / objective function and guardrails against functional regressions. It kind of closes the loop in that regard. One thing however is a test suite is not usually exhaustive in the sense that any code that passes the tests is valid. Usually tests are more complimentary in nature. Therefore you could still pos…

> One thing however is a test suite is not usually exhaustive in the sense that any code that passes the tests is valid. Usually tests are more complimentary in nature. Not in the world of AI - if your tests don't catch any known issues, the problem is the tests aren't comprehensive enough. There's no excuse at this point not to have an incredibly comprehensive test suite, to go with your other agent feedback loop co…

>> if your tests don't catch any known issues, the problem is the tests aren't comprehensive enough.

Maybe I misunderstand but this seems like a fairly low bar in the test suite only covers existing bugs.

I'd argue that if you aren't going to look at the code you actually need a fully comprehensive test suite - in the sense that if the tests pass, the code is correct and you don't have to look at it at all. The problem is, that isn't very quick to create it seems. Of course, if there is a way to do it quickly in a way that is reproducible by others I'd love to hear about it.

Post reply on HN