Probabilistic Flakiness: How do you test your tests? • Hacker News Board

Re: Probabilistic Flakiness: How do you test your tests?

#31

Post by singingsolitue » Tue, Dec 22, 2020, 1:07 PM UTC

justinpombrio wrote: post #6

_untra_ wrote: post #4

I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…

Yes, definitely that. I've also seen entire piles of tests be flaky because: - They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree. The general trend I've seen is that the more "e2e" a test is, the flakier it is.

End to end tests definitely end up being flaky, especially in large systems. One level of testing would be unit tests, but e2e tests have their own place, where they do end to end sanity checks. In my experience at Rippling, we have managed to identify a lot of such flakiness by pure first principles reasoning of the behavior, and in most cases, it turned out to be a subtle bug in the code. As the org grows larger, there should be a team that just attacks flaky tests, either from a fix point of view by reviewing tests, or from tools point of view, where finding the gap becomes easier for the product teams!

Re: Probabilistic Flakiness: How do you test your tests?

#32

Post by stiglitz » Tue, Dec 22, 2020, 4:42 PM UTC

“the right question to ask is not whether a particular test is flaky, but how flaky it is”

Yet again, a problem is not solved but paved over by a new level of indirection, and the house of cards grows taller.

Re: Probabilistic Flakiness: How do you test your tests?

#33

Post by vidoc » Tue, Dec 22, 2020, 5:06 PM UTC

> the functionality or reliability of our products for billions of people around the world

They can't help it, each facebook engineering blog post has to start with that scale porn.

Re: Probabilistic Flakiness: How do you test your tests?

#34

Post by hinkley » Tue, Dec 22, 2020, 8:15 PM UTC

_untra_ wrote: post #4

I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…

The worst thing to deal with in regard to determining correctness of code is global, shared, mutable state. Timestamps fit that bill (even though it's the Universe changing the state).

For testing purposes I often find myself making functions that take the date as an argument. If the language supports default values, I'll set it as a default value. If it doesn't, I'll make a convenience method or a null check to set it to 'now' if none is provided.

It turns out though that a lot of code we write to run within ±30 seconds of 'now' ends up over time having to run (or re-run) on old or future dates. So with the exception of logging and events, having that as an argument turns out to be useful or at least neutral.

For logging and events I'd probably use a mock timing library anyway.

Re: Probabilistic Flakiness: How do you test your tests?

#35

Post by hinkley » Tue, Dec 22, 2020, 8:19 PM UTC

rectang wrote: post #10

ddevault wrote: post #7

Earlier quoted context omitted.

Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.

But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.

They say that contempt is the beginning of the end of the Rule of Law, which is why you should be careful not to pass frivolous laws.

Tests I've found are much the same way. You don't write one flaky test and stop. If you write one and everybody is okay with it, you and your coworkers write more, and more, until there are 50, 100. Once the suite flakes out on an interval, nobody takes a failed test seriously, and then broken code doesn't get checked for hours. One broken test? I bet it's the usual. I'll just rerun it a couple of times.

The thread you're pulling on here starts to unravel the whole Continuous Integration sweater.

Re: Probabilistic Flakiness: How do you test your tests?

#36

Post by troelsSteegin » Thu, Dec 24, 2020, 1:22 PM UTC

How is measuring or estimating flakiness different than measuring or estimating variance? To me, something that is not flaky never errors for no reason. Something that is flaky errors infrequently and not obviously why. Something that is broken errors a lot. So, to me, flakiness is stochastic variation in a recurring process /with/ some ambiguity about the sources of variation. Flakiness make sense to me as a term for describing variation in relation to cause. That's different than just variance - or maybe flakiness is just variance? I am wondering what the FB framework looks like from a statistical process control POV.