We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs
1–6 of 6 posts
Re: We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs
#2Exact the same bullpoints and trade offs I am facing. More code, faster, more bugs, esp. more security bugs.
Part of the more bugs is that the tools find much more bugs and write much better tests than before with only fuzzing disturbing our peace for a while.
Re: We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs
#3Exact the same bullpoints and trade offs I am facing. More code, faster, more bugs, esp. more security bugs. Part of the more bugs is that the tools find much more bugs and write much better tests than before with only fuzzing disturbing our peace for a while.
We also find that the agents tend to find false positives. Especially when it comes to security. In theory, you can get an automated pen test every week. In practice, it drowns you in triaging reported vulnerabilities, as you find a significant part of them aren't something that you'd like to address, like ever.
Re: We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs
#4Exact the same bullpoints and trade offs I am facing. More code, faster, more bugs, esp. more security bugs. Part of the more bugs is that the tools find much more bugs and write much better tests than before with only fuzzing disturbing our peace for a while.
We also find that the agents tend to find false positives. Especially when it comes to security. In theory, you can get an automated pen test every week. In practice, it drowns you in triaging reported vulnerabilities, as you find a significant part of them aren't something that you'd like to address, like ever.
What models do you use for automated pen tests? Don't the Frontier labs block security stuff?
Re: We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs
#5Exact the same bullpoints and trade offs I am facing. More code, faster, more bugs, esp. more security bugs. Part of the more bugs is that the tools find much more bugs and write much better tests than before with only fuzzing disturbing our peace for a while.
We also find that the agents tend to find false positives. Especially when it comes to security. In theory, you can get an automated pen test every week. In practice, it drowns you in triaging reported vulnerabilities, as you find a significant part of them aren't something that you'd like to address, like ever.
Yes, I have the same. Then I have to clear memory and adjust AGENTS.md (symlinked to CLAUDE.md). Or delete the work. Happens rarely though. Like 5%.
In very rare cases it insists to being wrong. Then I have to switch models.
Re: We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs
#6Earlier quoted context omitted.
We also find that the agents tend to find false positives. Especially when it comes to security. In theory, you can get an automated pen test every week. In practice, it drowns you in triaging reported vulnerabilities, as you find a significant part of them aren't something that you'd like to address, like ever.
What models do you use for automated pen tests? Don't the Frontier labs block security stuff?
Opus was pretty decent at that before the whole Fable drama.
In fact, for our work, which is absolutely not rocket science most of the time, we see little upside in using Fable. The output is still "good enough," but the token burnout rate is through the roof. For many tasks, we stick to Opus, or even Sonnet, and the output is just fine.
Fun fact, we recently prepared an AI recruitment task. The basic idea was that there are conflicting goals in the context. We assumed that AI would lead candidates to a dead end, and they'd have to figure out what was happening. We abandoned the idea as even Sonnet was handling it fine enough. And it burned way fewer tokens than Fable would.