Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…
Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…
Provide agents with automated feedback
81–87 of 87 posts
Re: Provide agents with automated feedback
#82Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…
https://github.com/shepherdjerred/scout-for-lol/tree/main/es...
Ex here to detect duplicated code: https://github.com/shepherdjerred/scout-for-lol/blob/main/es...
Re: Provide agents with automated feedback
#83Earlier quoted context omitted.
Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…
OP isn't claiming all runtime bugs can be prevented with static lints suggested by LLMs but, if at least some can, I don't see how your comment is contributing. Yet another case of "your suggestion isn't perfect so I'll dismiss it" in Hacker News. Why is this such a common occurrence here? Does this fallacy have a name? EDIT: seems to be https://en.wikipedia.org/wiki/Nirvana_fallacy
My LLM has theorized that its success at answering trivia questions has left some people feeling threatened.
Re: Provide agents with automated feedback
#84Linters...custom made pre-commit linters which are aligned with your code base needs. The agents are great at creating these linters and then forevermore it can help feedback and guide them. My key repo now has "audit_logging_linter, auth_response_linter, datetime_linter, fastapi_security_linter, fastapi_transaction_linter, logger_security_linter, org_scope_linter, service_guardrails_linter, sql_injection_linter, tes…
Re: Provide agents with automated feedback
#85Earlier quoted context omitted.
> Right now i spent a lot of “back pressure” on fitting the scope of the task into something that will fit in one context window (ie the useful computation, not the raw token count). I suspect we will see a large breakthrough when someone finally figures out a good system for having the llm do this. I've found https://github.com/obra/superpowers very helpful for breaking the work up into logical chunks a subagent can…
Still basically relies on feeding context through natural language instructions which can be ignored or poorly followed? The answer is not more natural language guardrails, it is in (progressive) formal specification of workflows and acceptance criteria. The task cannot be marked as complete if it is only accessible through an API that rejects changes lacking proof that acceptance criteria were met.
However some specification only exists in natural language. IE: make this page optimized for a smartphone. The task of turning that vague direction into formal requirements is work in and of itself. The more you can have the llm help with that — the more time it will save you.
Re: Provide agents with automated feedback
#86Linters...custom made pre-commit linters which are aligned with your code base needs. The agents are great at creating these linters and then forevermore it can help feedback and guide them. My key repo now has "audit_logging_linter, auth_response_linter, datetime_linter, fastapi_security_linter, fastapi_transaction_linter, logger_security_linter, org_scope_linter, service_guardrails_linter, sql_injection_linter, tes…
Aren't many of those tests? Why define them as linters?
The key differences are:
1. Static vs Runtime Analysis
Linters use AST parsing to analyze code structure without executing it. Tests verify actual runtime behavior. Example from our datetime_linter:
tree = ast.parse(file_path.read_text())
for node in ast.walk(tree):
if isinstance(node, ast.Import):
if alias.name == "datetime":
# Violation: should use pendulum
This catches import datetime syntactically. A test would need to actually execute code and observe wrong datetime behavior.
2. Feedback Loop Speed
- Linters: Run in pre-commit hooks. Agent writes code → instant feedback → fix → iterate in seconds
- Tests: Run in CI. Commit → push → wait minutes/hours → fix in next session
For AI agents, this is critical. A linter that blocks commit keeps them on track immediately rather than discovering violations after a test run.
3. Structural Violations
For example, our `fastapi_security_linter` catches things like "route missing TenantRouter decorator". These are structural violations - "you forgot to add X" - not "X doesn't work correctly." Tests verify the behavior of X when it exists.
4. Coverage Exhaustiveness
Linters scan all code paths structurally. Tests only cover scenarios you explicitly write. Our org_scope_linter catches every unscoped platform query across the entire codebase in one pass. Testing that would require writing a test for each query.
5. The Hybrid Value
We actually have both. The linter catches "you forgot the security decorator" instantly. The test (test_fastapi_authorization.py) verifies "the security decorator actually blocks unauthorized users at runtime." Different failure modes, complementary protections.
Think of it like: linters are compile-time checks, tests are runtime checks. TypeScript catches string + number at compile time; you don't write a test for that.Re: Provide agents with automated feedback
#87Earlier quoted context omitted.
How would you compare it to Claude Code in planning mode?
I've only used Claude's planning mode when I just started using Claude Code, so it may be me using it wrong at the time, but the superpowers are way more helpful for picking up on you wanting to build/modify something and helping you brainstorm interactively to a solid spec, suggesting multiple options when applicable. This results in a design and implementation doc and then it can coordinate subagents to implement t…