Live data from Hacker News

Provide agents with automated feedback

banay.me

51–60 of 87 posts

Re: Provide agents with automated feedback

#51

As someone said: Custom lints are super useful. What we do at https://minfx.ai (a Neptune/Wandb replacement) is we use TONS of custom lints. Anytime we see some undesireable repeatable agent behavior, we add it as a prompt modification and a lint. This is relatively easy to do in Rust. The kinds of things I did are: - Specify maximum number of lines / tabs, otherwise code must be refactored. - Do not use unsafe or Re…

What is DAG ordering of structs?

Re: Provide agents with automated feedback

#52

I got turned off in the first paragraph with the misuse of the term "back pressure". "back pressure" is a term from data engineering to specifically indicate a feedback signal that indicates a service is overloaded and that clients should adapt their behavior. Backpressure != feedback (the more general term). And in the agentic world, we use the term 'context' to describe information used to help LLMs make decisions,…

the back pressure terminology comes from me. essentially it’s the wheel - you need to add backpressure to the agentic flywheel.

see https://ghuntley.com/pressure

i have the pleasure to work with moss and he came up with a way to explain what is in my head with ease.

Re: Provide agents with automated feedback

#53

As someone said: Custom lints are super useful. What we do at https://minfx.ai (a Neptune/Wandb replacement) is we use TONS of custom lints. Anytime we see some undesireable repeatable agent behavior, we add it as a prompt modification and a lint. This is relatively easy to do in Rust. The kinds of things I did are: - Specify maximum number of lines / tabs, otherwise code must be refactored. - Do not use unsafe or Re…

What is DAG ordering of structs?

Each struct and its referenced fields can be thought of as a graph which can be sorted. Ideally, it is a DAG, but sometimes you can have recursive structures so it can be a cyclic graph. By DAG-ordering a I meant a topological sorting such that you do it by layers of the graph.

https://en.wikipedia.org/wiki/Topological_sorting

https://en.wikipedia.org/wiki/Directed_acyclic_graph

Re: Provide agents with automated feedback

#54

As someone said: Custom lints are super useful. What we do at https://minfx.ai (a Neptune/Wandb replacement) is we use TONS of custom lints. Anytime we see some undesireable repeatable agent behavior, we add it as a prompt modification and a lint. This is relatively easy to do in Rust. The kinds of things I did are: - Specify maximum number of lines / tabs, otherwise code must be refactored. - Do not use unsafe or Re…

What is DAG ordering of structs?

DAG is directed acyclic graph. A bit like a tree where branches are allowed to merge but there are no cycles.

Re: Provide agents with automated feedback

#55

Earlier quoted context omitted.

What is DAG ordering of structs?

DAG is directed acyclic graph. A bit like a tree where branches are allowed to merge but there are no cycles.

Yes, but I was wondering how organize your code in a DAG.

Re: Provide agents with automated feedback

#56

Earlier quoted context omitted.

> Right now i spent a lot of “back pressure” on fitting the scope of the task into something that will fit in one context window (ie the useful computation, not the raw token count). I suspect we will see a large breakthrough when someone finally figures out a good system for having the llm do this. I've found https://github.com/obra/superpowers very helpful for breaking the work up into logical chunks a subagent can…

How would you compare it to Claude Code in planning mode?

I've only used Claude's planning mode when I just started using Claude Code, so it may be me using it wrong at the time, but the superpowers are way more helpful for picking up on you wanting to build/modify something and helping you brainstorm interactively to a solid spec, suggesting multiple options when applicable. This results in a design and implementation doc and then it can coordinate subagents to implement the different features, followed by spec review and code review. Really impressed with it, I use it for anything non-trivial.

Re: Provide agents with automated feedback

#57
post #30
post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

I like this idea but I can’t think of a concrete example to ground it. Can anybody share a real example?

I got tired of brittle literals like `http://localhost:3000` and `postgres://…@127.0.0.1/...` creeping into my code, so I wrote a few ESLint rules that detect “hardcoded infrastructure” strings and ask the agent to find the constant in the codebase — not by guessing its name but by `grep`-ing for its value.

The detection is based on dumb string literal heuristics, but has proven rather effective. Example patterns:

const hardcodedInfrastructure = { url: /^https?:\/\/(localhost|127\.0\.0\.1|192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)(:\d+)?/i, dbUrl: /^(postgresql|postgres|mysql|mongodb):\/\/.*@(localhost|127\.0\.0\.1|192\.168\.\d+\.\d+|10\.\d+\.\d+\.\d+|172\.(1[6-9]|2\d|3[01])\.\d+\.\d+)/i, localhost: /^localhost$/i, localhostPort: /^localhost:\d+$/i, };

Re: Provide agents with automated feedback

#58
post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it".

To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if "is variable XYZ unused?" or "does variable X oveshadow Y" or other more "esoteric" lint rules will not catch it. Likelihood is high that the LLM just hallucinated some false positive lint rule anyways giving you a false sense of security.

Re: Provide agents with automated feedback

#59
post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…

> static code analysis which by definition does not find runtime bugs

I'm not sure if there's some subtlety of language here, but from my experience of javascript linting, it can often prevent runtime problems caused by things like variable scoping, unhandled exceptions in promises, misuse of functions etc.

I've also caught security issues in Java with static analysis.

Re: Provide agents with automated feedback

#60
post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

I disagree. A ton of linting is cosmetic stuff, you need more specific guardrails, even though linting in general is helpful.
Post reply on HN