A new era for software testing • Hacker News Board

Re: A new era for software testing

#41

Post by aplomb1026 » Thu, Jun 11, 2026, 9:35 PM UTC

dkn wrote: post #20

mlmonkey wrote: post #16

Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D

I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.

[flagged]

Re: A new era for software testing

#42

Post by rahoulb » Thu, Jun 11, 2026, 10:05 PM UTC

konart wrote: post #14

simianwords wrote: post #4

Scenario testing is the new word for it and I think this is a game changer. Two of the reasons I never liked writing tests is - they didn’t seem to usually assert much internal logic - they would have to be maintained along with the original code I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot. So imagine I’m making an…

>Scenario testing is the new word How is scenario different from a behavior (as in Behavior-Driven Development)? Gherkin and things like Cucumber are not something new, are they?

My clankers are instructed to use "Outside In development" with "red/green TDD" at all times.

They write really good Gherkin features and then work inwards writing unit tests as they go - checking that they fail before implementation so it's actually testing something worthwhile.

And the code they ship is decent quality (not as good as me most of the time - but a LOT better than me when I'm tired or I'm pissed off about something or the work is really boring).

Re: A new era for software testing

#43

Post by spaceclay » Thu, Jun 11, 2026, 10:07 PM UTC

avensec wrote: post #24

> The idea is to create a markdown file where an AI agent is asked to work as a QA engineer Given your code-base is mature enough, please don't have a single Skill/Steering/Persona/Ruleset (or whatever) for your "QA Engineer." This is just the same "my behavioral file can one-shot the entire system build" kind of thinking that will give you expensive, marginal results as the system grows. If you want to have success…

[dead]

Re: A new era for software testing

#44

Post by kovek » Thu, Jun 11, 2026, 10:50 PM UTC

mlmonkey wrote: post #16

Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D

I heard people say this before. I'm wondering, how do you instruct the LLM to generate the tests? Do you tell it the scenarios that would be covered, or do you just tell it to write tests for the code?

Re: A new era for software testing

#45

Post by 8note » Thu, Jun 11, 2026, 10:56 PM UTC

onlyrealcuzzo wrote: post #40

achierius wrote: post #34

Earlier quoted context omitted.

Hate to be a pedant, but that's really not what "zero cost abstractions" means. The idea behind those is that you get a cleaner interface to some gross machine functionality/OS API/etc. layer, but don't pay a performance cost vs. using the gross lower-level layer. E.g. Rust's Option, unlike C++'s std::optional. What you're thinking of is "no runtime" or "lightweight runtime", which does often mean "no garbage collect…

Rust's zero cost abstractions mainly stem from its affine ownership model managing memory lifetimes safely and correctly with zero cost - as that is the killer feature... That's what I do. When people think of "zero cost" they don't think about std::optional. They think about not having to manage memory lifetimes AND NOT having to pay for a Garbage Collector to do it for you. That was always the trade you made until…

> affine ownership

huh? you can rotate and scale the ownership?

Re: A new era for software testing

#46

Post by kulahan » Thu, Jun 11, 2026, 11:05 PM UTC

Isn’t this explicitly the one place you’d never want to use AI? Like, the only actual problem with AI is that it sometimes ignores errors in output like it has a PHD in Blindness To Problems. I always figured the path forward was strictly enforced and managed tests written by hand, because who gives a shit about the code behind it as long as you can prove that the output is real?

Ten million blackboxes with ten billion tests or whatever. Otherwise it’s literally the blind leading the blind

Re: A new era for software testing

#47

Post by simoncion » Thu, Jun 11, 2026, 11:12 PM UTC

Daishiman wrote: post #36

acdha wrote: post #19

Earlier quoted context omitted.

Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.

Agents read the docs. People don't. That's the underlying reason.

> People don't.

People falling all over themselves to write docs for their pile-of-linear-algebra-with-a-smiley-face-painted-on-it [0] don't read the docs, no. People who give a shit about writing solid software that doesn't get them paged at three in the damn morning do.

[0] The face is there to provide social-trustworthiness signals to engage the human pack-bonding instinct, natch.

Re: A new era for software testing

#48

Post by dragonwriter » Thu, Jun 11, 2026, 11:20 PM UTC

righthand wrote: post #32

anthonypasq wrote: post #13

Earlier quoted context omitted.

are we just re-inventing playwright tests except 10x slower and infinity times more expensive? i feel like im going insane

Well playwright tests used to be called puppeteer tests which used to be called selenium tests, so you tell me.

Ťhose are all technology variations of “automated web ui tests”, which is a subset of “automated ui tests”, which is itself almost (but not quite exactly) a subset of “automated user acceptance tests”, none of which are new categories.

Re: A new era for software testing

#49

Post by dragonwriter » Thu, Jun 11, 2026, 11:21 PM UTC

simianwords wrote: post #29

acdha wrote: post #19

Earlier quoted context omitted.

Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.

[flagged]

Positing an equivalence between a dismissive term for AI bots and a racial slur against black people is, like, super racist.

Re: A new era for software testing

#50

Post by acdha » Fri, Jun 12, 2026, 12:12 AM UTC

simianwords wrote: post #29

acdha wrote: post #19

Earlier quoted context omitted.

Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.

[flagged]

That’s a rather stunning comparison: racism is a problem because it’s unfairly treating sentient beings but a pile of linear algebra is not even sentient, much less your peer. That’s part of why I used the term: “agent” isn’t current because agents have, well, agency and can be held accountable.

https://lucumr.pocoo.org/2026/5/26/clankers/