Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A new era for software testing

antirez.com

Post reply on HN
Print view View original

61–67 of 67 posts

Re: A new era for software testing

#61

Post by wavemode »

onlyrealcuzzo wrote: post #33
rglover wrote: post #15

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly. Nothing has changed about the discipline o…

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead. I can tell you rig…

You're not wasting your time; LLMs have written plenty of compilers. Compilers are easy for LLMs to work on, because their level of verifiability is very high. That is, an LLM can easily determine whether what a compiler is doing is correct or incorrect.

Automated verifiability goes down once a software project incorporates things like:

- Concurrency

- Networking / distributed systems

- Visuals / animations

- Domain knowledge (e.g. banking, finance)

Re: A new era for software testing

#62

Post by onlyrealcuzzo »

wavemode wrote: post #61
onlyrealcuzzo wrote: post #33

Earlier quoted context omitted.

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead. I can tell you rig…

You're not wasting your time; LLMs have written plenty of compilers. Compilers are easy for LLMs to work on, because their level of verifiability is very high. That is, an LLM can easily determine whether what a compiler is doing is correct or incorrect. Automated verifiability goes down once a software project incorporates things like: - Concurrency - Networking / distributed systems - Visuals / animations - Domain…

Yes and no, they are easy to verify they work sometimes, hard to verify they work all times.

Re: A new era for software testing

#64

Post by ptx »

Is the use of the term "automatic programming" a deliberate parallel to the development of COBOL [1]?

If so, if this is meant to imply that LLMs are just another step towards higher-level abstractions, the analogy doesn't quite work. Unlike a COBOL compiler, the LLMs output can't be predicted or reasoned about, so you can't really fix bugs in your program (i.e. your prompt) but only try to permute it haphazardly and hope for the best.

[1] https://ethw.org/Milestones:A-0_Compiler_and_Initial_Develop...

Re: A new era for software testing

#65

Post by pydry »

mlmonkey wrote: post #16

Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D

If your tests align with the spec and the tooling is good it isnt tedious.

If you find writing tests tedious enough to make using an LLM to write them seem like a good idea you're probably churning out repetitive tests, unnecessary tests, tests which aren't great at catching bugs.

Re: A new era for software testing

#66

Post by ahartmetz »

marshalhq wrote: post #21

I ran mutation testing on a side project recently and found a test that passed even if the production method returned an empty string. AI-generated tests at scale will have exactly this problem. High coverage, confident test names, zero actual verification.

IME there are these levels of tests:

- If you call the setter, the getter returns the same value - these are kinda bullshit and would be caught by the next level anyway

- Testing basic normal use

- Testing known difficulties of the implementation

- Exhaustive or randomized (if necessary) testing of the state space, ~= property-based testing

I expect AI to have very different levels of ability for these, not necessarily in strictly descending order as listed.

Re: A new era for software testing

#67

Post by simianwords »

skydhash wrote: post #39
simianwords wrote: post #3

Earlier quoted context omitted.

I think this is just TDD or unit test dogma and I’m personally not a fan. Unit tests and deterministic tests are hard to get right and need to be done at the correct boundary. I have seen many people dogmatically pushing unit tests religiously but this often leads to very hard to maintain tests that mostly exist just to change along with the main code itself. A good way to understand if your unit tests are good: are…

> A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak. Unit tests are great for pure algorithms, like file format, data encoding, crypto,… etc. Everything with a specs that will rarely changes. You write your tests once and basically never have to update them. But for requirem…

> But for requirements that changes often like in a enterprise settings or applications, maintaining a suite of unit tests is expensive. Integration tests are better because contracts between modules don’t change that much. Even if the suite are not exhaustive, they’re useful enough to catch some failures.

Yes this is what I'm trying to say.

Post reply on HN