[flagged]
The "just retry" approach is truly bothersome. I think it is at least partly an organizational issue, because it happens far more often when QA is a separate team.
The purpose of continuous integration is to fail
31–40 of 52 posts
Re: The purpose of continuous integration is to fail
#32Earlier quoted context omitted.
> IMO, good tests are relatively immutable. You should be able to have multiple valid implementations. You should add new tests to describe the new functionality of that implementation, however, the old tests should remain relatively untouched. Taken to extreme this would mean getting rid of unit tests altogether in favor of functional and/or end-to-end testing. Which is... a strategy. I don't know if it is a good or…
If you can't tell, I actually think functional tests have a lot more value than most unit tests :) Kent Dodd agrees with me. [1] This isn't to say I see no value in unit tests, just that they should tend towards describing the function of the code under test, not the implementation. [1] https://kentcdodds.com/blog/the-testing-trophy-and-testing-c...
The goal of unit tests is to circumvent problems with performance or specificity from functional tests.
If you haven't seen those problems with yours, unit tests would be useless.
Re: The purpose of continuous integration is to fail
#33> Whenever a CI run fails, we can re-run it. If it passes the second time, we are sure it was flaky. Or you have a concurrency issue in your production code?
Then the test is still flaky. If there's a bug you want the test to consistently fail, not just sometimes.
The parent is talking about when the implementation is flaky, not the test. When you go to fix the problem under that scenario there is no reason for you to modify the test. The test is fine.
Re: The purpose of continuous integration is to fail
#34> Whenever a CI run fails, we can re-run it. If it passes the second time, we are sure it was flaky. Or you have a concurrency issue in your production code?
Then the test is still flaky. If there's a bug you want the test to consistently fail, not just sometimes.
But also a flaky test is a bug by itself.
Re: The purpose of continuous integration is to fail
#35> One dreaded and very common situation is when a failing CI run can be made to pass by simply re-running it. We call this flaky CI.
> Flaky CI is nasty because it means that a CI failure no longer reliably indicates that a mistake was caught. And it is doubly nasty because it is unfixable (in theory); sometimes machines just explode.
> Luckily flakiness can be detected: Whenever a CI run fails, we can re-run it. If it passes the second time, we are sure it was flaky.
One of the specialties that I have (unwillingly!) specialized in at my current company is CI flakes. Nearly all flakes, well over 90% of them, are not "unfixable", nor are they even really some boogy man unreliable thing that can't be understood.
The single biggest change I think we made that helped was having our CI system record the order¹ in which tests are run. Rerunning the tests, in the same order, makes most flakes instantly reproduce locally. Probably the next biggest reproducer is "what was the time the test ran?" and/or running it in UTC.
But once you get from "it's flakey" (and fails "seeming" "at" "random") to "it fails 100% of the time on my laptop when run this way" then it becomes easier to debug, b/c you can re-run it, attach a debugger, etc. Database sort issues (SQL is not deterministically ordered unless you ORDER BY), issues with database IDs (e.g., test expects row ID 3, usually gets row ID 3, but some other test has bumped us to row ID 4²), timezones — those are probably the biggest categories of "flakes".
While I know what people express with "flake", "flake" as a word is usually "failure mode I don't understand".
(Excluding truly transitory issues like a network failure interfering with a docker image pull, or something.)
(¹there are a lot of reasons people don't have deterministically ordered CI runs. Parallelism, for example. Our order is deterministic, b/c we made a value judgement that random orderings introduce too much chaos. But we still shard our tests across multiple VMs, and that sharding introduces its own changes to the order, as sometimes we rebalance one test to a different shard as devs add or remove tests.)
²this isn't usually because the ID is hardcoded, it is usually b/c, in the test, someone is doing `assert Foo.id == Bar.id`, unknowningly. (The code is usually not straight-forward about what the ID is an ID to.) I call this ID type confusion, and it's basically weakly-typed IDs in langs where all IDs are just some i32 type. FooId and BarId types would be better, and if I had a real type system in my work's lang of choice…
Re: The purpose of continuous integration is to fail
#36This is stupidly obvious but you'd be surprised how many people have the attitude that competent developers should have tested their code manually before making PRs so you shouldn't need CI.
Re: The purpose of continuous integration is to fail
#37> One dreaded and very common situation is when a failing CI run can be made to pass by simply re-running it. We call this flaky CI. > Flaky CI is nasty because it means that a CI failure no longer reliably indicates that a mistake was caught. And it is doubly nasty because it is unfixable (in theory); sometimes machines just explode. > Luckily flakiness can be detected: Whenever a CI run fails, we can re-run it. If…
> those are probably the biggest categories of "flakes".
Interesting. In my experience, it is always either a concurrency issue in the program under test or PBTs finding some extreme edge case that was never visited before.
Re: The purpose of continuous integration is to fail
#38[flagged]
If a build fails 10% of the time, it actually takes 100x longer before to fail for the 10%x10%x10% case.
Re: The purpose of continuous integration is to fail
#39This is of course true as a blanket "gotcha" headline- although I wouldn't call a failed test the CI itself failing. A real failure would be a false positive, a pass where there wasn't coverage, or a failure when there was no breaking change. Covering all of these edge cases can become as tiresome as maintaining the application in the first place (of course this is a generalization)
100% coverage is an EXPTIME problem.
Re: The purpose of continuous integration is to fail
#40Earlier quoted context omitted.
Then the test is still flaky. If there's a bug you want the test to consistently fail, not just sometimes.
The parent is talking about when the implementation is flaky, not the test. When you go to fix the problem under that scenario there is no reason for you to modify the test. The test is fine.
What you're describing is the every day reality but what you WANT is that if your implementation has a race condition, then you want a test that 100% of the time detects that there is a race condition (rather than 1% of the time).