Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI is forcing us to write good code

bits.logic.inc

Post reply on HN
Print view View original

221–229 of 229 posts

Re: AI is forcing us to write good code

#221

Post by melozo »

I’m not sure how controversial this is - but 100% code coverage is almost always a waste of time, paid both immediately and long term, for certain languages. Go, for example, requires explicit error handling, but the way errors are handled are usually plain and homogenous. Adding unit testing everywhere creates a phenomenal amount of test code that can become 3x the size of the source, and certain changes (like interface changes) can require updates to all tests, especially if mocking is used.

Obviously with AI maybe those issues I have go away. But I really don’t like letting the AI modify tests without meticulously manually reviewing those changes, because in my experience the AI cares more about getting the tests passing than it does about ensuring semantic correctness. For as long as tests are manually maintained I will continue keeping them as few as necessary while maintaining what I view as an acceptable amount of coverage.

Re: AI is forcing us to write good code

#222

Post by godelski »

ben_w wrote: post #160
KurSix wrote: post #134

There's a catch with 100% coverage. If the agent writes both the code and the tests, we risk falling into a tautology trap. The agent can write flawed logic and a test that verifies that flawed logic (which will pass). 100% coverage only makes sense if tests are written before the code or rigorously verified by a human. Otherwise, we're just creating an illusion of reliability by covering hallucinations with tests. A…

All the problems you list are true, but the solutions not so much. I've seen this problem with humans even back at university when it was the lecturer's own example attempting to illustrate the value of formal methods and verification. I would say the solution is neither "get humans to do it" nor "do it before writing code", but rather "get multiple different minds involved to check each other's blind spots, and no m…

  > "get multiple different minds involved to check each other's blind spots
This is actually my big gripe about chatbot coding agents. They are trained on human preference and thus they optimize for errors that are in our blind spots.

I don't think people take this subtly seriously enough. Unless we have an /objective/ ground truth we end up proxying our optimization. So we don't optimize for code that /is/ correct, we optimize for code that /looks/ correct. It may seem like a subtle difference but it is critical.

The big difference is when they make errors they are errors that are more likely to be difficult for humans to detect.

Good tools should complement tool users. Fill in gaps. But as we've been trying to train agents to replace humans we are not focusing on this distinction. I want my coding agent to make errors that are obvious to me just as I want errors I make to be obvious to it (or for it to be optimized to detect errors I make)

Re: AI is forcing us to write good code

#223

Post by eternityforest »

KurSix wrote: post #134

There's a catch with 100% coverage. If the agent writes both the code and the tests, we risk falling into a tautology trap. The agent can write flawed logic and a test that verifies that flawed logic (which will pass). 100% coverage only makes sense if tests are written before the code or rigorously verified by a human. Otherwise, we're just creating an illusion of reliability by covering hallucinations with tests. A…

Tests freeze behavior in place, and manual end to end testing can confirm that the most common paths are at least kind of correct ish.

Obviously that's not good enough, but I'd much rather have AI tests than poor test coverage.

Re: AI is forcing us to write good code

#224

Post by godelski »

vrighter wrote: post #173
joshribakoff wrote: post #172

Earlier quoted context omitted.

That’s why you’ve gotta test your tests. Insert bugs and ensure they fail. As the sibling comments alluded to, it’s not exclusively an AI problem since multiple people can miss the issue too. It’s wonderful that AI is an impetus for so many people to finally learn proper engineering principles though!

but who will test the tests of tests?

Just add tests to test your test tests

Re: AI is forcing us to write good code

#225

Post by hoppp »

I prefer to write critical code and ask the llm for snippets like if I was googling docs, then I got maximum guard rails, it can't alter the project by itself, all code is reviewed by me and I can refactor as I go .

If Iam building guard rails to let the LLM directly code then I am building guard rails and not the project I want, I don't want to babysit an LLM, I just want to get on with my work.

I also don't agree with the title, a very prominent new dev community called vibe coders emerged and they are all about low quality code created fast. So LLMs mostly write bad code.

Re: AI is forcing us to write good code

#226

Post by zweifuss »

zweifuss wrote: post #151
cloudhead wrote: post #148

Earlier quoted context omitted.

What's MC/DC?

Modified Condition/Decision Coverage It's mandated by DO-178C for the highest-level (Level A) avionics software. Example: if (A && B || C) { ... } else { ... } needs individual tests for A, B, and C. Test #,A,B,A && B,Outcome taken,Shows independence for 1,True,True,True,if branch,(baseline true) 2,False,True,False,else branch,A (A flips outcome while B fixed at True) 3,True,False,False,else branch,B (B flips outcome…

I made a mistake:

  Test #  A      B      C      Result
  1       True   True   False  True
  2       False  True   False  False
  3       True   False  False  False
  4       False  True   True   True

Re: AI is forcing us to write good code

#227

Post by [deleted] »

zweifuss wrote: post #151
cloudhead wrote: post #148

Earlier quoted context omitted.

What's MC/DC?

Modified Condition/Decision Coverage It's mandated by DO-178C for the highest-level (Level A) avionics software. Example: if (A && B || C) { ... } else { ... } needs individual tests for A, B, and C. Test #,A,B,A && B,Outcome taken,Shows independence for 1,True,True,True,if branch,(baseline true) 2,False,True,False,else branch,A (A flips outcome while B fixed at True) 3,True,False,False,else branch,B (B flips outcome…

[deleted]

Re: AI is forcing us to write good code

#228

Post by KurSix »

godelski wrote: post #222
ben_w wrote: post #160

Earlier quoted context omitted.

All the problems you list are true, but the solutions not so much. I've seen this problem with humans even back at university when it was the lecturer's own example attempting to illustrate the value of formal methods and verification. I would say the solution is neither "get humans to do it" nor "do it before writing code", but rather "get multiple different minds involved to check each other's blind spots, and no m…

> "get multiple different minds involved to check each other's blind spots This is actually my big gripe about chatbot coding agents. They are trained on human preference and thus they optimize for errors that are in our blind spots. I don't think people take this subtly seriously enough. Unless we have an / objective / ground truth we end up proxying our optimization. So we don't optimize for code that / is / correc…

Exactly, we've basically trained an army of perfect corporate suck-ups. AI optimizes for the "least resistance at review" metric, not "logic correctness," so relying on eyeballs to check AI code right now is risky, we need "unfeeling" validators: compilers, formal verification, and rigid tests

Re: AI is forcing us to write good code

#229

Post by KurSix »

joshribakoff wrote: post #172
KurSix wrote: post #134

There's a catch with 100% coverage. If the agent writes both the code and the tests, we risk falling into a tautology trap. The agent can write flawed logic and a test that verifies that flawed logic (which will pass). 100% coverage only makes sense if tests are written before the code or rigorously verified by a human. Otherwise, we're just creating an illusion of reliability by covering hallucinations with tests. A…

That’s why you’ve gotta test your tests. Insert bugs and ensure they fail. As the sibling comments alluded to, it’s not exclusively an AI problem since multiple people can miss the issue too. It’s wonderful that AI is an impetus for so many people to finally learn proper engineering principles though!

Mutation testing is becoming the only way to catch AI red-handed. Without mutations you'll be staring at a perfect CI/CD dashboard, unaware that your tests verify absolutely nothing

Yeah, it burns CPU like crazy, but CPU time is dirt cheap right now compared to the cost of an engineer debugging that self-deception in production

Post reply on HN