Live data from Hacker News

Provide agents with automated feedback

banay.me

81–87 of 87 posts

Re: Provide agents with automated feedback

#81
post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…

Static analysis certainly can find runtime bugs

Re: Provide agents with automated feedback

#82
post #24

Y'all are sleeping on custom lint rules. Every time you find a runtime bug, ask the LLM if a static lint rule could be turned on to prevent it, or have it write a custom rule for you . Very few of us have time to deep dive into esoteric custom rule configuration, but now it's easy. Bonus: the error message for the custom rule can be very specific about how to fix the error. Including pointing to documentation that ex…

I have examples of this here for eslint

https://github.com/shepherdjerred/scout-for-lol/tree/main/es...

Ex here to detect duplicated code: https://github.com/shepherdjerred/scout-for-lol/blob/main/es...

Re: Provide agents with automated feedback

#83
post #77

Earlier quoted context omitted.

Nobody is sleeping on anything. Linting for the most part is static code analysis which by definition does not find runtime bugs. You even say it yourself "runtime bug, ask the LLM if a static lint rule could be turned on to prevent it". To find most runtime bugs (e.g. incorrect regex, broken concurrency, incorrect SQL statement, ...) you need to understand the mental model and logic behind the code - finding out if…

OP isn't claiming all runtime bugs can be prevented with static lints suggested by LLMs but, if at least some can, I don't see how your comment is contributing. Yet another case of "your suggestion isn't perfect so I'll dismiss it" in Hacker News. Why is this such a common occurrence here? Does this fallacy have a name? EDIT: seems to be https://en.wikipedia.org/wiki/Nirvana_fallacy

Well, if you haven't noticed, LLM topics receive a particularly hostile reaction on HN.

My LLM has theorized that its success at answering trivia questions has left some people feeling threatened.

Re: Provide agents with automated feedback

#84

Linters...custom made pre-commit linters which are aligned with your code base needs. The agents are great at creating these linters and then forevermore it can help feedback and guide them. My key repo now has "audit_logging_linter, auth_response_linter, datetime_linter, fastapi_security_linter, fastapi_transaction_linter, logger_security_linter, org_scope_linter, service_guardrails_linter, sql_injection_linter, tes…

Aren't many of those tests? Why define them as linters?

Re: Provide agents with automated feedback

#85

Earlier quoted context omitted.

> Right now i spent a lot of “back pressure” on fitting the scope of the task into something that will fit in one context window (ie the useful computation, not the raw token count). I suspect we will see a large breakthrough when someone finally figures out a good system for having the llm do this. I've found https://github.com/obra/superpowers very helpful for breaking the work up into logical chunks a subagent can…

Still basically relies on feeding context through natural language instructions which can be ignored or poorly followed? The answer is not more natural language guardrails, it is in (progressive) formal specification of workflows and acceptance criteria. The task cannot be marked as complete if it is only accessible through an API that rejects changes lacking proof that acceptance criteria were met.

Some specification exists as formal constraints. Ie: c code will or will not compile.

However some specification only exists in natural language. IE: make this page optimized for a smartphone. The task of turning that vague direction into formal requirements is work in and of itself. The more you can have the llm help with that — the more time it will save you.

Re: Provide agents with automated feedback

#86
post #84

Linters...custom made pre-commit linters which are aligned with your code base needs. The agents are great at creating these linters and then forevermore it can help feedback and guide them. My key repo now has "audit_logging_linter, auth_response_linter, datetime_linter, fastapi_security_linter, fastapi_transaction_linter, logger_security_linter, org_scope_linter, service_guardrails_linter, sql_injection_linter, tes…

Aren't many of those tests? Why define them as linters?

Great question, I let Claude help answer this...see below:

The key differences are:

  1. Static vs Runtime Analysis                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  Linters use AST parsing to analyze code structure without executing it. Tests verify actual runtime behavior. Example from our datetime_linter:                                                                                                                                                                
                                                                                                                                                                                                                                                                                                                 
  tree = ast.parse(file_path.read_text())                                                                                                                                                                                                                                                                        
  for node in ast.walk(tree):                                                                                                                                                                                                                                                                                    
      if isinstance(node, ast.Import):                                                                                                                                                                                                                                                                           
          if alias.name == "datetime":                                                                                                                                                                                                                                                                           
              # Violation: should use pendulum                                                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                 
  This catches import datetime syntactically. A test would need to actually execute code and observe wrong datetime behavior.                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                 
  2. Feedback Loop Speed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  - Linters: Run in pre-commit hooks. Agent writes code → instant feedback → fix → iterate in seconds                                                                                                                                                                                                            
  - Tests: Run in CI. Commit → push → wait minutes/hours → fix in next session                                                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                                                 
  For AI agents, this is critical. A linter that blocks commit keeps them on track immediately rather than discovering violations after a test run.                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  3. Structural Violations
  For example, our `fastapi_security_linter` catches things like "route missing TenantRouter decorator". These are structural violations - "you forgot to add X" - not "X doesn't work correctly." Tests verify the behavior of X when it exists.

  4. Coverage Exhaustiveness                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
  Linters scan all code paths structurally. Tests only cover scenarios you explicitly write. Our org_scope_linter catches every unscoped platform query across the entire codebase in one pass. Testing that would require writing a test for each query.                                                        
                                                                                                                                                                                                                                                                                                                 
  5. The Hybrid Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
  We actually have both. The linter catches "you forgot the security decorator" instantly. The test (test_fastapi_authorization.py) verifies "the security decorator actually blocks unauthorized users at runtime." Different failure modes, complementary protections.                                         
                                                                                                                                                                                                                                                                                                                 
  Think of it like: linters are compile-time checks, tests are runtime checks. TypeScript catches string + number at compile time; you don't write a test for that.

Re: Provide agents with automated feedback

#87

Earlier quoted context omitted.

How would you compare it to Claude Code in planning mode?

I've only used Claude's planning mode when I just started using Claude Code, so it may be me using it wrong at the time, but the superpowers are way more helpful for picking up on you wanting to build/modify something and helping you brainstorm interactively to a solid spec, suggesting multiple options when applicable. This results in a design and implementation doc and then it can coordinate subagents to implement t…

I asked because I started using GSD which I liked at first, but have since dropped. I started using planning instead and find it probably does a better job and is waaay faster. After a while of using GSD I started to realize the control I initially felt over the model with all these markdown documents (current state, phases, phases belonging to larger milestones) etc. were an illusion.
Post reply on HN