No one who claims to automate testing ever seems to have a sophisticated understanding of testing. I suspect the urge people have for automating anything comes not from the love and skill they possess for that activity, but rather for the imagined fruits of it. Software testing is easy to fake, which means it’s particularly easy to claim automation accomplished. Therefore, all these people thinking they are going to…
What's the gap between a Co-pilot written test and a human-written test? Are tests written by co-pilot better than no tests? Developers generally hate writing tests, so I suspect more and more unit tests will be generated. I've worked with someone who does this and it seems insane to me, although I never got to see the generated tests they were referring to, so I can't give specifics.
The next problem is that ChatGPT is not thinking critically about risk. It thinks (well, no it doesn't think... let's say it "treats")... it treats testing as a process of demonstration. Demonstration is not testing. Demonstration proves that functionality is possible, but not that it is reliable. Thus, it produces checking code that is shallow.
The next problem is how are you going to explain to ChatGPT what your product does? Do you give it all your source code? Do you give it all your Jira tickets? The demos I have seen are toy examples. Nothing on a realistic scale.
Let's say you feed it a whole spec and source code, somehow. The next problem is omissions. ChatGPT arbitrarily stops producing output. You then have to carefully check its work to see what it has left out.
What you are asking ChatGPT to do, really, is to write code that will maximize the probability of spotting a real bug, while minimizing the probability of a false positive. But it will arbitrarily focus on only those kinds of bugs that it can easily discover. This is the oracle problem.
The next problem is that ChatGPT halucinates or misunderstands, so you have to correct its mistakes. Sometimes going through frustrating iterations of prompts, like a man trying to pull a mule through town.
You also have trouble with test data and data setup. Only in toy examples is this not a significant problem.
ChatGPT produces conventional results, and that does have some value. But it's not enough for professional work.
I just think the people who casually say that ChatGPT is going to easily handle these things are pretty stupid, under the definition of stupidity as "the refusal to think."