[dead]
Provide agents with automated feedback
71–80 of 87 posts
Re: Provide agents with automated feedback
#72Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…
Anthropic ships an official plugin to create linters for you based on your Claude Code history or instructions, it’s great. You can vibe code your lint rules per repo.
https://github.com/anthropics/claude-code/tree/main/plugins/...
Re: Provide agents with automated feedback
#73Appropriate feedback is critical for good long horizon performance. The direction of feedback doesn't necessarily have to be from autonomous tools back to the LLM. It can also flow from tools to humans who then iterate the prompt / tools accordingly. I've recently discovered that if a model gets stuck in a loop on a tool call across many different runs, it's almost certainly because of a gap in expectations regarding…
ReportToolIssue… holy shit that is genius.
Re: Provide agents with automated feedback
#74Earlier quoted context omitted.
> static code analysis which by definition does not find runtime bugs I'm not sure if there's some subtlety of language here, but from my experience of javascript linting, it can often prevent runtime problems caused by things like variable scoping, unhandled exceptions in promises, misuse of functions etc. I've also caught security issues in Java with static analysis.
The usefulness of using static code analysis (strict type systems, linting) versus not using static code analysis is out of the question. Specifically JavaScript which does not have a strict type system benefits greatly from using static code analysis. But the author claims that you can catch runtime bugs by letting the LLM create custom lint rules, which is hyperbole at least and wrong at most and giving developers…
> But the author claims that you can catch runtime bugs
I think you misinterpreted OP:
Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it
Key word is prevent.
Re: Provide agents with automated feedback
#75Geoffrey Huntley uses the analogy of a "pottery wheel" and says "everything is a ralph loop".
Re: Provide agents with automated feedback
#76Earlier quoted context omitted.
The usefulness of using static code analysis (strict type systems, linting) versus not using static code analysis is out of the question. Specifically JavaScript which does not have a strict type system benefits greatly from using static code analysis. But the author claims that you can catch runtime bugs by letting the LLM create custom lint rules, which is hyperbole at least and wrong at most and giving developers…
> But the author claims that you can catch runtime bugs I think you misinterpreted OP: Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it Key word is prevent .
Catch or prevent - linting only covers a tiny (depending on programming language sometimes more sometimes less) subset of runtime problems. The whole back pressure discussion feels like AI coders found out about type systems and lint rules - but it doesn’t resolve the type problems we get in agentic coding. The only „agent“ responsible for code correctness (and thus adherence to feature specification) is the human instructing the agent, a better compiler or lint rule will not prevent massive logic bugs LLMs tend to create like tests testing functions that have been created by the LLM for the test to make it pass, broken logic flows, missing DI, recreating existing logic, creating useless code that’s not being used anywhere yet pollutes context windows - all the problems LLM based „vibe“ coding „shines“ with once you work on a sufficiently long running project.
Why do I care so much about this? Because the „I feel left behind“ crowd is being gaslighted by comments like the OPs.
Overall strict type systems and static code analysis have always been good for programming, and I‘m glad vibe coders are finding out about this as well - it just doesn’t fix the lack of intelligence LLMs have nor the responsibility of programmers to understand and improve the generated stochastic token output
Re: Provide agents with automated feedback
#77Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…
Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…
OP isn't claiming all runtime bugs can be prevented with static lints suggested by LLMs but, if at least some can, I don't see how your comment is contributing. Yet another case of "your suggestion isn't perfect so I'll dismiss it" in Hacker News.
Why is this such a common occurrence here? Does this fallacy have a name?
EDIT: seems to be https://en.wikipedia.org/wiki/Nirvana_fallacy
Re: Provide agents with automated feedback
#78I’m looking forward to when people realize that the agents stay more focused when feedback is provided more frequently, with adjustments to the spec made after every round of feedback, i.e., agile.
Re: Provide agents with automated feedback
#79Earlier quoted context omitted.
I like this idea but I can’t think of a concrete example to ground it. Can anybody share a real example?
Claude Code is obsessed with using single letter names for inline function parameters and as loop control variables. I don't like it and I think it is sloppy, so I told it to stop in CLAUDE.md. In my experience, Claude Code will respect CLAUDE.md around 70 % of the time, it seems to cherry pick areas that it will respect more and less often and of course it kept ignoring this instruction. So I told it to add a pre-co…
Nice. The big picture strategy is figuring out the right mix of deterministic and probabilistic programming elements and how to mix them. I see agents ignore clear instructions all the time thus we need to codify the important instructions into deterministic rules, which probably take the form of tests or linters.
Re: Provide agents with automated feedback
#80Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…
I agree, I have 'written' a handful of rubocop rules that are hyper specific to the codebase I work on. I never would have bothered before claude code. Stuff like using out custom logger correctly, or to not use Rails.env because we have our own (weird of course) env system.