Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Provide agents with automated feedback

banay.me

Post reply on HN
Print view View original

61–70 of 87 posts

Re: Provide agents with automated feedback

#61

Post by cube2222 »

achou wrote: post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

Yeah, I have written multiple almost completely-vibecoded linters since Claude Code came out, and they provide very high value.

It’s kind of a best case scenario use-case - linters are generally small and easy to test.

It’s also worth noting that linters now effectively have automagical autofix - just run an agent with “fix the lints”. Again, one of the best case scenarios, with a very tight feedback loop for the agent, sparing you a large amount of boring work.

Re: Provide agents with automated feedback

#62

Post by epolanski »

This article is sensible but I'd argue it states the obvious.

The back pressure I need cannot come from automated testing or access to an LSP.

The back pressure I need comes from following rules it has been given, or listening to architectural or business logic feedback.

On that, I still cannot make it work like I want. Going to provide a simple example with Claude Codes.

I have a frontend agent instructed to not use any class or style ever, only the design system components and primitives.

Not only it will ignore those very quickly, but when it proposes edits and I give feedback the agent ignores them completely and instead it keeps suggesting more edits.

Thus I had to revert to deleting the agent completely and rely on the main thread for doing that work.

Same applies with any other agent.

Re: Provide agents with automated feedback

#63

Post by epolanski »

jamesblonde wrote: post #37

I got turned off in the first paragraph with the misuse of the term "back pressure". "back pressure" is a term from data engineering to specifically indicate a feedback signal that indicates a service is overloaded and that clients should adapt their behavior. Backpressure != feedback (the more general term). And in the agentic world, we use the term 'context' to describe information used to help LLMs make decisions,…

I find your kind of comment pointless.

The term back pressure actually comes from mechanical engineering in the context of steam engines.

It first appeared in a dictionary 160 years ago.

Words are just words. Mathematicians very well understand that words mean nothing, what matters are definitions and the author provides one.

E.g. natural numbers may or may not contain the number 0, but that's irrelevant, because what mathematicians care for are definitions, so they will state that natural numbers are a given a set of positive whole numbers (including or not the number 0) and avoid arguing about labels. You can call them funky numbers or neet numbers, doesn't matter.

Same applies here. Your comment is pointless because the author does provide a definition for back pressure in the context of his blog post and what matters is discussing the concept he labels in the context of LLMs.

Re: Provide agents with automated feedback

#64

Post by jkhdigital »

AnonyX387 wrote: post #35
qazxcvbnmlp wrote: post #15

My mental model is that ai coding tools are machines that can take a set of constraints and turn them into a piece of code. The better you get at having it give its self those constraints accurately, the higher level task you can focus on. Eg compiler errors, unit tests, mcp, etc. Ive heard of these; but havent tried them yet. https://github.com/hmans/beans https://github.com/steveyegge/gastown Right now i spent a lo…

> Right now i spent a lot of “back pressure” on fitting the scope of the task into something that will fit in one context window (ie the useful computation, not the raw token count). I suspect we will see a large breakthrough when someone finally figures out a good system for having the llm do this. I've found https://github.com/obra/superpowers very helpful for breaking the work up into logical chunks a subagent can…

Still basically relies on feeding context through natural language instructions which can be ignored or poorly followed?

The answer is not more natural language guardrails, it is in (progressive) formal specification of workflows and acceptance criteria. The task cannot be marked as complete if it is only accessible through an API that rejects changes lacking proof that acceptance criteria were met.

Re: Provide agents with automated feedback

#65

Post by aeneas_ory »

normie3000 wrote: post #59
aeneas_ory wrote: post #58

Earlier quoted context omitted.

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…

> static code analysis which by definition does not find runtime bugs I'm not sure if there's some subtlety of language here, but from my experience of javascript linting, it can often prevent runtime problems caused by things like variable scoping, unhandled exceptions in promises, misuse of functions etc. I've also caught security issues in Java with static analysis.

The usefulness of using static code analysis (strict type systems, linting) versus not using static code analysis is out of the question. Specifically JavaScript which does not have a strict type system benefits greatly from using static code analysis.

But the author claims that you can catch runtime bugs by letting the LLM create custom lint rules, which is hyperbole at least and wrong at most and giving developers a false sense of security at worst.

Re: Provide agents with automated feedback

#66

Post by yorwba »

actionfromafar wrote: post #55
tablatom wrote: post #54

Earlier quoted context omitted.

DAG is directed acyclic graph. A bit like a tree where branches are allowed to merge but there are no cycles.

Yes, but I was wondering how organize your code in a DAG.

Identifiers correspond to nodes and a mention of an identifier in the definition of another corresponds to a directed edge. The resulting graph won't necessarily be acyclic, but you can still use it to inform the order in which you present definitions, e.g. newspaper style starts with the most high-level function and puts the low-level details at the end: https://pypi.org/project/flake8-newspaper-style/

Re: Provide agents with automated feedback

#67

Post by sandGorgon »

achou wrote: post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

could you explain what this does ? like i ask it to write a testcase (pytest, etc). why is a linter better ?

Re: Provide agents with automated feedback

#68

Post by michalsustr »

yorwba wrote: post #66
actionfromafar wrote: post #55

Earlier quoted context omitted.

Yes, but I was wondering how organize your code in a DAG.

Identifiers correspond to nodes and a mention of an identifier in the definition of another corresponds to a directed edge. The resulting graph won't necessarily be acyclic, but you can still use it to inform the order in which you present definitions, e.g. newspaper style starts with the most high-level function and puts the low-level details at the end: https://pypi.org/project/flake8-newspaper-style/

Yes exactly! Good idea to extend it to functions as well

Re: Provide agents with automated feedback

#69

Post by tomashubelbauer »

oofbey wrote: post #30
achou wrote: post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

I like this idea but I can’t think of a concrete example to ground it. Can anybody share a real example?

Claude Code is obsessed with using single letter names for inline function parameters and as loop control variables. I don't like it and I think it is sloppy, so I told it to stop in CLAUDE.md. In my experience, Claude Code will respect CLAUDE.md around 70 % of the time, it seems to cherry pick areas that it will respect more and less often and of course it kept ignoring this instruction. So I told it to add a pre-commit hook and invoke the TypeScript compiler and analyze the AST for single-letter variable names and tank the pre-commit check when it detects one with an error message indicating the offending symbols' locations. Now it can be non-deterministic as much as it wants, but it will never commit this particular flair of slop again as the adherence is verified deterministically. I already have a few more rules in mind I want to codify this way to prevent it from reproducing patterns it was trained on that I don't like and consider low quality.

Re: Provide agents with automated feedback

#70

Post by garganzol »

I find this article profoundly insightful. On a side note, the text reminds me the good old days of internet, where everybody shared useful information without strings attached. No attention seeking, no ads, no emotional drama. Just spot on perfect
Post reply on HN