What we do when we detect a failure is freeze the test runner instance, and allocate the same failed test to another runner. If the second runner succeeds, we okay the build, but we put the test and the frozen runner in a queue for investigation, and some engineer will be responsible for diagnosing and fixing this intermittent test. This queue is worked every day on a rotating schedule. We run our own CI/CD infrastru…
Has this strategy worked perfectly so far? I would imagine a best two of three method would work better like those used for integrity in high radiation environments. However, I'm guessing code is significantly more deterministic and the danger of false positives (tests passing when they shouldn't) is much smaller.
False positives are also super costly for us, as everyone works on trunk (by design, to avoid skew) and deploys all ckeckins directly to production. "Best out of two" is sufficient for the old tests, and if someone creates new intermittent tests, we follow up with education so that isn't a persistent problem.