Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Probabilistic Flakiness: How do you test your tests?

engineering.fb.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Probabilistic Flakiness: How do you test your tests?

#21

Post by vlovich123 »

jaggederest wrote: post #12
pfdietz wrote: post #9

Earlier quoted context omitted.

Mutation testing is a way to test your tests.

And will very quickly humble even the most diligent of testers :) For example, you're testing that this value is in a range - but are you testing both endpoints and whether they're inclusive or exclusive? In faster languages I enjoy quickcheck style testing which also reveals these things pretty quickly.

Do not confuse end-to-end functional tests that this article is describing for smaller “unit” tests that you typically might express in quick check/mutation testing (at least as I understand the space).

Re: Probabilistic Flakiness: How do you test your tests?

#22

Post by 10x-dev »

AnonC wrote: post #18
10x-dev wrote: post #2

This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…

> FB approaches internal code development with the same data driven rigor as their business decisions. How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature. I have a few anecdotes where, as a Facebook group…

My point was the tools are there and are pretty advanced, compared to what I see available outside of the company.

Whether an internal team uses those tools effectively depends entirely on them and their expertise.

Re: Probabilistic Flakiness: How do you test your tests?

#23

Post by boulos »

This note from the article jumped out at me:

> While engineers tend to trust passing test results, they often retry failing tests a number of times on the same version of code and consider failures followed by passing results as flaky. We do not have a good theoretical explanation for why this behavior prevails.

which seems to me like standard confirmation bias.

When writing a test or updating code, you clearly assume you got it right. So a passing test “confirms” that.

Personally, I find myself quite susceptible to this bias when updating code (i.e., when I haven’t broken any tests, I assume I updated my code correctly) but I’m much more cautious when writing new tests. I usually prefer to write a failing test, and get it to pass, to mitigate some of that bias (but fully admit that once it turns green, I’m overly confident at the “clearly correct” nature of the output).

Re: Probabilistic Flakiness: How do you test your tests?

#24

Post by deckard1 »

justinpombrio wrote: post #6
_untra_ wrote: post #4

I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…

Yes, definitely that. I've also seen entire piles of tests be flaky because: - They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree. The general trend I've seen is that the more "e2e" a test is, the flakier it is.

e2e is the number one productivity killer at my org, by a long shot. If there is such a thing as a non-flaky e2e test, I have yet to see it. That, or the test does nothing.

Selenium seems fine for the most part. It's a solid tool. Where it falls apart is when developers do not account for all the various ways the browser session will go wrong. There are simply too many variables at play. A/B tests, cookies, popups, network conditions, machine speed (and current load). Writing e2e is like being blindfolded and tying one hand behind your back while your coworkers take turns spinning you in your chair.

It constantly amazes me that companies think they can put more tasks on a developer's plate with zero impact to productivity. They really do believe that all testing is free and will somehow pay for itself. And yet a single QA human going through a simple testing plan will catch more actual bugs than hundreds of e2e tests that cost a literal fortune to maintain.

Re: Probabilistic Flakiness: How do you test your tests?

#25

Post by 10x-dev »

chii wrote: post #19
10x-dev wrote: post #2

This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…

> tie up all these loose ends without giving up your personal time. so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.

Seeing as I made the backlog, I have nobody but myself to blame :)

I know what you mean though. Our deadlines are self-imposed, but the planning process is there to make sure that the effort remains high.

I believe the managers are responsible for squeezing out as much juice as they can in a polite way (as opposed to other places where they were pretty direct about it).

I suppose the combination of opportunity + compensation justifies the stress level and excitement at this point in my life.

I've worked at other places where people primarily coasted with low pay but received good retirement benefits. It seemed to work for them.

Re: Probabilistic Flakiness: How do you test your tests?

#26

Post by hinkley »

10x-dev wrote: post #2

This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…

I kinda miss working on teams where disabling the tests was plan B instead of plan G. The disabled tests should hopefully reduce code coverage. Also someone should be graphing out the volume and flux in disabled tests from week to week, to tap the brakes if it looks like we're rushing forward blindly instead of doing our due diligence.

Re: Probabilistic Flakiness: How do you test your tests?

#27

Post by marshmellman »

slaymaker1907 wrote: post #15
rectang wrote: post #10

Earlier quoted context omitted.

But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.

We actually have different classes of tests to allow for some more flaky tests. You definitely don't want to run those flaky tests after every build, but you should be able to eventually get a run where those tests pass before handing it off to customers.

Interesting. Doesn’t this imply that the behavior itself is flaky?

Re: Probabilistic Flakiness: How do you test your tests?

#28

Post by vvwvvwvvw »

AnonC wrote: post #18
10x-dev wrote: post #2

This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…

> FB approaches internal code development with the same data driven rigor as their business decisions. How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature. I have a few anecdotes where, as a Facebook group…

They sure have a lot of rigor in sabotaging ad blocking efforts. If anything their anti anti ablock works all the time.

Re: Probabilistic Flakiness: How do you test your tests?

#29

Post by pydry »

justinpombrio wrote: post #6
_untra_ wrote: post #4

I've found the number one cause of flakiness in tests is from misuse of nondeterministic or highly stateful functions. This becomes especially apparent when you recognize a lot of nondeterminism comes from misusing datetime libraries such as momentJS, or from Math.random . Even integration tests against certain "eventually consistent" databases tends to result in flakey tests. Entropy and Time libraries are intention…

Yes, definitely that. I've also seen entire piles of tests be flaky because: - They relied on something or other over the network, or - They used Selenium to try to see whether the Dom was updated correctly, and either Selenium itself or the way the code interacted with it made _all_ the tests flaky to some degree. The general trend I've seen is that the more "e2e" a test is, the flakier it is.

IME the source of flakiness in tests is always something that you can deal with given enough time and about 15-20% of the time it is a bug in the code itself, sometimes quite a dangerous one.

Re: Probabilistic Flakiness: How do you test your tests?

#30

Post by ddevault »

rectang wrote: post #10
ddevault wrote: post #7

Earlier quoted context omitted.

Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.

But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.

Right - but you fix the bug. You don't add a tolerance.
Post reply on HN