This is a great example of using statistical inference to solve a problem at scale. However I had one question: Since this using Bayesian inference, how was the prior over the parameters p_f and p_b determined? I could not tell from reading the article.
Probabilistic Flakiness: How do you test your tests?
11–20 of 36 posts
Re: Probabilistic Flakiness: How do you test your tests?
#12Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
Mutation testing is a way to test your tests.
And will very quickly humble even the most diligent of testers :) For example, you're testing that this value is in a range - but are you testing both endpoints and whether they're inclusive or exclusive? In faster languages I enjoy quickcheck style testing which also reveals these things pretty quickly.
Re: Probabilistic Flakiness: How do you test your tests?
#13Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
Somehow you build a feedback loop of tests.
Re: Probabilistic Flakiness: How do you test your tests?
#14Hmm. One has to wonder how do one test the test of your tests, then. And those tests of the tests of my tests too...
The final test is always human review. The point of testing is to simplify the analysis of correctness for humans.
Also, ideally your code and test are somewhat independently written, whether it is whitebox or blackbox. If that is the case,you greatly reduce the chance that both the code and the test are wrong in such a way that the test still passes.
This is why writing unit tests for stuff like getters and setters is really dumb. It doesn't really simplify checking whether the code is correct unless you use something like meanbean and thus provides some degree of separation between the test and the implementation.
Re: Probabilistic Flakiness: How do you test your tests?
#15Earlier quoted context omitted.
Agreed. Flaky tests are a bug and the only agreeable solution is to identify and remove the non-deterministic inputs.
But sometimes the bug is in the code not the test, and you wouldn't have known about the bug if you didn't write the flaky test! A flaky test which fails once in every N test suite runs is better than no test at all.
We actually have different classes of tests to allow for some more flaky tests. You definitely don't want to run those flaky tests after every build, but you should be able to eventually get a run where those tests pass before handing it off to customers.
Re: Probabilistic Flakiness: How do you test your tests?
#16This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…
> FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever. This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
Not sure what you mean. There is no hard and fast top down enforced rules. Aside from natural Hack bias at FB, you can use any language you want. People develop tools in Rust, D, C++, bash, etc.
As far as libraries go, there is a mandate to use in-house patched versions of various sdks and libraries, for security/customization reasons. Otherwise, go nuts.
People write a whole lot of in-house software at FB, some of which is later open sourced.
> That just doesn't happen at Amazon
Oh please. We had a mandate to use Apollo.
PHP is banned at Amazon (whatever your view of PHP is, that is a top down directive).
The only freedom we actually had at Amazon was to use whatever open source software we could without contributing back to it (I'm sure there are examples where we did, ut not at the scale of FB).
And lastly, Amazon was all about telling me what to do. One day I got a phone call and I got moved from front-end app development to backend c++. Nobody asked me if I wanted to do it or if I would be any good at it.
FB spends months in bootcamp with teams trying to convince you to join them.
Re: Probabilistic Flakiness: How do you test your tests?
#17This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…
> FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest hardware request ever. This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use. That just doesn't happen at Amazon.
> This is a natural consequence and trade-off of having a top-down directive of which tools/languages/frameworks to use
That's not a natural consequence of the structure.
If the top-down directives are issued by competent and financially accountable leadership, you get a nice efficiency pipeline.
Re: Probabilistic Flakiness: How do you test your tests?
#18This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…
> FB approaches internal code development with the same data driven rigor as their business decisions.
How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature.
I have a few anecdotes where, as a Facebook group administrator of some large (tens of thousands of members) and small groups (few thousand members), there were so many bugs in moderating the group with an added mega bug where the “Report issue to Facebook” (or something to that effect) would also throw an error. Those bugs lasted years (going into the current year) before I gave up and quit Facebook.
Based on my experience over several years, I doubt there’s much rigor on many things. It may just be one of those marketing messages that have a little bit of truth and a lot of subterfuge.
Re: Probabilistic Flakiness: How do you test your tests?
#19This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…
> tie up all these loose ends without giving up your personal time.
so this just means they've crammed more into your backlog than you can do. I would not give up any personal, unpaid time to do it, unless there's some promotion that you need which this is a demonstration/investment for.
Re: Probabilistic Flakiness: How do you test your tests?
#20This is just one of the reasons why I found FB infrastructure to be great. I used to work at Amazon before moving to FB and the difference in internal tool quality is night and day. FB approaches internal code development with the same data driven rigor as their business decisions. Everything is measured, so I knew for example that the slow build times I was experiencing were slower than 99% of everyone else. Easiest…
> FB approaches internal code development with the same data driven rigor as their business decisions. How does this even matter if the end result is buggy for years and there’s nobody to alert? It seems like Facebook has many versions of code that get deployed in some places and not others, never attempting to reach an eventual convergence even for the same feature. I have a few anecdotes where, as a Facebook group…
In Facebook Ad Manager it is similarly hard to perform any task without having your day ruined.