> There is never a “Nothing found, good to go!”. I have to keep reminding myself that the model will always give me what I ask for, regardless of the reality/truth.
Tell it something like:
Before doing any commits or producing a summary for the user, you must run a verification sub-agent.
Its goal is to adversarially and critically check your supposed findings to look out for false positives and hallucinations.
Doing so with a separate sub-agent with relatively clean context (but with all the relevant details of the problem space that appear to be facts) should improve our confidence in the findings.
Maybe also something like:
Try to classify each found issue as either SERIOUS, CRITICAL or NITPICK, discard nitpicks, we only care about impactful issues.
It should
somewhat cut down on the useless output.
I've largely found the same in regards to generating code - the initial pass will often have bugs that the model itself can find but only when run as a separate sub-agent without the confidence poisoning in its own previous output.