1. The automated system works on testing multiple configs on multiple branches. When problems arise they are marked in the testing system.
2. In the really good ones a bug can then be filed in the issue tracker that includes one or more "rules". These "rules" cover the failures discoverd in the testing system, with limits on the matrix of configs, branches, and time ranges.
3. So back in the testing system you can distinguish between "covered" and "new" bugs. You also can immediately see when a bug has been fixed, as the system then points out tests that were expected to fail that are not suceeding (flaky tests are a really nasty problem in this).
4. In an ideal system (I was working on this part at one point) the failures would not simply be of the "it failed" variety, but rather look for a fingerprint (e.g.: patterns in the logged output, or in the stack-traces). Then you could see when another bug encoches on the first (sadly common in really large systems), or when two bugs happen to hit the same tests on different configs at about the same time (again, sadly common), and even have the system suggest when old problems re-emmerge to expand the rule coverage.
Sadly there are no testing systems out there (that I have found) that can really support the "configs" model to make this really worth it. Filling that hole is what I would probably do if I won the lottery... but since I have to make a living...