I'm sympathetic to the viewpoint that GPT-4 is prone to mistakes when writing code. Unfortunately, the analysis in this paper is pretty bad and doesn't support that conclusion.
The authors assume that for any given method under consideration, it must only occur within a particular pattern of other method calls and control flow instructions. But the templates they have chosen are clearly only applicable in certain situations.
For example, they claim that I/O operations are "wrong" unless they are wrapped in exception handlers that log any errors:
try {
...
} catch (IOException e) {
e.printStackTrace();
}
But of course, this will cause execution to continue as though the I/O was successful, which might be exactly the wrong thing to do! In many cases, you
want the exception to propagate, so that the caller can decide how to handle the failure. (And even if you do want to report the error somehow, writing it to stderr might not be correct; it's pointless in a GUI app.)
Similarly, the authors assume that every time you create a file or directory, you always want to call .exists() first (even though doing so has an inherent race condition); that Map.get() must always be followed by an "if" block; that List.get() must always be guarded by an explicit bounds check; that after doing a database query, you always want to close the connection; and so on. None of those rules are universally applicable.
I would expect the real problem with LLM-generated code to be semantic bugs and "misunderstandings" of the requirements, which would not be caught by superficial checks like this.