I ran mutation testing on a side project recently and found a test that passed even if the production method returned an empty string. AI-generated tests at scale will have exactly this problem. High coverage, confident test names, zero actual verification.
A new era for software testing
21–30 of 67 posts
Re: A new era for software testing
#22I ran mutation testing on a side project recently and found a test that passed even if the production method returned an empty string. AI-generated tests at scale will have exactly this problem. High coverage, confident test names, zero actual verification.
Don't worry, AI maximalists have a solution: create tests for the tests.
Re: A new era for software testing
#23Scenario testing is the new word for it and I think this is a game changer. Two of the reasons I never liked writing tests is - they didn’t seem to usually assert much internal logic - they would have to be maintained along with the original code I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot. So imagine I’m making an…
So, throw out the traditional test pyramid, shift right, and rely more on persona testing than fine-grained atomic tests? I would hope teams don't need to re-learn that lesson for themselves, but...
Re: A new era for software testing
#24> The idea is to create a markdown file where an AI agent is asked to work as a QA engineer
Given your code-base is mature enough, please don't have a single Skill/Steering/Persona/Ruleset (or whatever) for your "QA Engineer." This is just the same "my behavioral file can one-shot the entire system build" kind of thinking that will give you expensive, marginal results as the system grows.
If you want to have success in this space, get really fine-grained. Every single test scope needs its own behavioral files.
Have your core behavioral file define some simple specifics around Test Pyramid, Test Purposes, checks for tautological tests, etc. Then get _really_ specific;
-architect (plan)
-engineer (execute)
-resolver (problem solver, maintenance, how to manage a failure, etc.)
e.g., playwright-architect, etc.
Then create additional ones for Unit tests, API tests, contract tests, or any other required test layer for the SUT.
Overengineered? Maybe given the size of your codebase. But for anything significant, you are codifying what humans and their skillsets do.
Re: A new era for software testing
#25Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.
Same for me. I actively ask the LLM to write as few tests as possible. Otherwise you end up redundant and low value ttests.
Re: A new era for software testing
#26The idea of injecting more indeterminacy in pipelines is beyond me.
Re: A new era for software testing
#27The idea of injecting more indeterminacy in pipelines is beyond me.
Well you see, you just run the same test 10,000 times, and then...
Re: A new era for software testing
#28Earlier quoted context omitted.
I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.
Same for me. I actively ask the LLM to write as few tests as possible. Otherwise you end up redundant and low value ttests.
Yep, and wasted token spend on an ongoing basis.
I instruct the LLM to follow TDD practices in certain areas, but otherwise prioritize integration style tests at the edges.
Re: A new era for software testing
#29Earlier quoted context omitted.
since the rise of agentic coding tools, it feels like we're in a new "eternal september" of people discovering ui end-to-end test automation.
Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.
[flagged]
Re: A new era for software testing
#30Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code. But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.
I have found a decent trick to be to write a parameterized test with e.g. a `cases` array that tests a function how you want it tested. Then ask the LLM to fill out more cases. It’s not perfect, but results in much less brittleness since you’ve already defined the specifics of what gets tested and what doesn’t.