Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A new era for software testing

antirez.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: A new era for software testing

#31

Post by righthand »

simianwords wrote: post #4

Scenario testing is the new word for it and I think this is a game changer. Two of the reasons I never liked writing tests is - they didn’t seem to usually assert much internal logic - they would have to be maintained along with the original code I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot. So imagine I’m making an…

This already exists. You mean capturing user flows which should already be supplied by product to the developer. A decent system is Behavior Driven Development (though honestly a poor acronym for it’s use).

Re: A new era for software testing

#32

Post by righthand »

anthonypasq wrote: post #13
simianwords wrote: post #4

Scenario testing is the new word for it and I think this is a game changer. Two of the reasons I never liked writing tests is - they didn’t seem to usually assert much internal logic - they would have to be maintained along with the original code I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot. So imagine I’m making an…

are we just re-inventing playwright tests except 10x slower and infinity times more expensive? i feel like im going insane

Well playwright tests used to be called puppeteer tests which used to be called selenium tests, so you tell me.

Re: A new era for software testing

#33

Post by onlyrealcuzzo »

rglover wrote: post #15

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly. Nothing has changed about the discipline o…

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.

I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead.

I can tell you right now:

1) It's 100x more than I could have achieved with zero compiler design experience.

2) I'm HIGHLY skeptical that LLMs can build something of this complexity (in some ways it's more difficult than implementing a Rust compiler) - so the testing is quite robust - 3 different systems (unit, integration, fuzz tests) each with mutant testing, each with between ~65-90% line coverage and ~50-80% branch coverage, combined with ~99% line coverage and ~86% branch coverage.

There is ZERO chance I could get something even close to this level of "working" by myself ever - let alone with minimal effort.

The test is kind of simple - if LLM's can do this... They should be able to do just about anything... Compilers are notoriously difficult to verify they actually work, rather than just kind of work sometimes...

People can say I'm wasting my time all they want.

But, one, it's been enlightening. I'm literally in awe of what they can do and have done.

Two, I've developed a bunch of tooling / metrics necessary to get them to be able to do something at this level of complexity without falling over themselves. And I think it can work at scale pretty easily.

Nearly all of the research comes from the 80s or farther back for the complexity metrics.

Re: A new era for software testing

#34

Post by achierius »

onlyrealcuzzo wrote: post #33
rglover wrote: post #15

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly. Nothing has changed about the discipline o…

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead. I can tell you rig…

Hate to be a pedant, but that's really not what "zero cost abstractions" means. The idea behind those is that you get a cleaner interface to some gross machine functionality/OS API/etc. layer, but don't pay a performance cost vs. using the gross lower-level layer. E.g. Rust's Option, unlike C++'s std::optional.

What you're thinking of is "no runtime" or "lightweight runtime", which does often mean "no garbage collector".

Re: A new era for software testing

#35

Post by zerr »

mlmonkey wrote: post #16

Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D

Some companies (e.g. Microsoft) used to have "Software Engineers in Test" who's job was writing such tests all day long, so that those developers who were developing features wouldn't waste time on it.

Re: A new era for software testing

#36

Post by Daishiman »

acdha wrote: post #19
hugs wrote: post #17

Earlier quoted context omitted.

since the rise of agentic coding tools, it feels like we're in a new "eternal september" of people discovering ui end-to-end test automation.

Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.

Agents read the docs. People don't. That's the underlying reason.

Re: A new era for software testing

#37

Post by spaceclay »

dkn wrote: post #20
mlmonkey wrote: post #16

Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D

I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.

[dead]

Re: A new era for software testing

#38

Post by pfdietz »

onemoresoop wrote: post #22
marshalhq wrote: post #21

I ran mutation testing on a side project recently and found a test that passed even if the production method returned an empty string. AI-generated tests at scale will have exactly this problem. High coverage, confident test names, zero actual verification.

Don't worry, AI maximalists have a solution: create tests for the tests.

That's what mutation testing is.

Re: A new era for software testing

#39

Post by skydhash »

simianwords wrote: post #3
wrxd wrote: post #2

I believe this can work if done on top of traditional testing. I would feel very uneasy to replace deterministic (ok, not always but mostly) test suites with something that is not deterministic at all

I think this is just TDD or unit test dogma and I’m personally not a fan. Unit tests and deterministic tests are hard to get right and need to be done at the correct boundary. I have seen many people dogmatically pushing unit tests religiously but this often leads to very hard to maintain tests that mostly exist just to change along with the main code itself. A good way to understand if your unit tests are good: are…

> A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak.

Unit tests are great for pure algorithms, like file format, data encoding, crypto,… etc. Everything with a specs that will rarely changes. You write your tests once and basically never have to update them.

But for requirements that changes often like in a enterprise settings or applications, maintaining a suite of unit tests is expensive. Integration tests are better because contracts between modules don’t change that much. Even if the suite are not exhaustive, they’re useful enough to catch some failures.

Re: A new era for software testing

#40

Post by onlyrealcuzzo »

achierius wrote: post #34
onlyrealcuzzo wrote: post #33

Earlier quoted context omitted.

> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead. I can tell you rig…

Hate to be a pedant, but that's really not what "zero cost abstractions" means. The idea behind those is that you get a cleaner interface to some gross machine functionality/OS API/etc. layer, but don't pay a performance cost vs. using the gross lower-level layer. E.g. Rust's Option, unlike C++'s std::optional. What you're thinking of is "no runtime" or "lightweight runtime", which does often mean "no garbage collect…

Rust's zero cost abstractions mainly stem from its affine ownership model managing memory lifetimes safely and correctly with zero cost - as that is the killer feature... That's what I do.

When people think of "zero cost" they don't think about std::optional. They think about not having to manage memory lifetimes AND NOT having to pay for a Garbage Collector to do it for you. That was always the trade you made until Rust.

I add on some cost to locks to prevent deadlock, and some cost to loops to insert co-operative yields in concurrent contexts unless you turn it off.

Post reply on HN