Probabilistic Flakiness: How do you test your tests?
engineering.fb.com
Probabilistic Flakiness: How do you test your tests?
1–10 of 36 posts
Re: Probabilistic Flakiness: How do you test your tests?
#2This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day.
FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever.
So it should be no wonder that even tests are measured. Flaky tests get disabled. That's nice and keeps trust high in the test framework. The downside is that you are likely to forget to fix the disabled test, so now some functionality that was important enough to test in the first place becomes untested and leaves room for others to introduce regressions. This has happened in my team, but the pace at FB is so quick that there is no way to tie up all these loose ends without giving up your personal time. Or maybe it requires someone more diligent than me.
Either way, I just wanted to share that I personally found the idea of measuring test runs and acting on insights from thise measurements really powerful. It's a theme that can be found at all levels of FB infrastructure.
Re: Probabilistic Flakiness: How do you test your tests?
#3Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
Re: Probabilistic Flakiness: How do you test your tests?
#4I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intentionally nondeterministic, so the use of these should be recognized when writing tests, and designing functional "cores" that can do predictable operations when given testable inputs.
Ain't no one got time for flakey tests.
Re: Probabilistic Flakiness: How do you test your tests?
#5Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
tests of test tests are test tests, so just use your test testing tests to test your test tests
Re: Probabilistic Flakiness: How do you test your tests?
#6I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…
Yes, definitely that. I've also seen entire piles of tests be flaky because:
- They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree.
The general trend I've seen is that the more "e2e" a test is, the flakier it is.
Re: Probabilistic Flakiness: How do you test your tests?
#7I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…
Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.
Re: Probabilistic Flakiness: How do you test your tests?
#8This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…
> FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever.
This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
Re: Probabilistic Flakiness: How do you test your tests?
#9Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
Mutation testing is a way to test your tests.
Re: Probabilistic Flakiness: How do you test your tests?
#10I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…
Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.
But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.