Software testing, and why I'm unhappy about it
nhaehnle.blogspot.com
Software testing, and why I'm unhappy about it
1–10 of 76 posts
Re: Software testing, and why I'm unhappy about it
#2Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).
Re: Software testing, and why I'm unhappy about it
#3> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions.
And that's why we test and why tests shouldn't be allowed to fail.
Just because the scenarios described make testing hard does not change reality of what makes tests valuable.
If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based development and see if you like the waits and constant rebasing in large projects/teams. But don't eff with the bloody tests!
Re: Software testing, and why I'm unhappy about it
#4Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).
Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.
Re: Software testing, and why I'm unhappy about it
#5Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).
Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.
It also makes it impossible to test outside of the public API
Re: Software testing, and why I'm unhappy about it
#6Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).
Tests in seperate repo is the worst anti pattern I have seen. It’s extremely common that a change requires a change in tests but it’s impossible to correctly manage this situation if the tests can’t be updated in the same commit/pr.
The only time "tests in a separate repo" makes sense to me is if they are truly cross-functional end to end tests that exercise several systems.
Those tests should be as small as possible to verify that everything is still wired together correctly.
Everything else should be either unit tests or narrow integration tests between a small handful of components. And as you said, they should live in the repository of the software they test.
Re: Software testing, and why I'm unhappy about it
#7> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based…
when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code.
At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket will be created for the test owner to fix their test (and it will be disabled for future test runs)
I think for small to medium projects, you can have all tests succeed but once the repo is large enough / has frequent enough changes, flakey tests are bound to slip in.
Re: Software testing, and why I'm unhappy about it
#8> During day-to-day development, the important bit isn't that there are no failures. The important bit is that there are no regressions. And that's why we test and why tests shouldn't be allowed to fail. Just because the scenarios described make testing hard does not change reality of what makes tests valuable. If pre-existing failures are halting the production pipeline and you don't like it, switch off trunk based…
when the codebase gets large enough you need to allow some tests to "fail" but really I mean you need a way to quickly mark a failing test as flakey so the author can fix it while everyone else can get on with their day and merge code. At $dayjob this works well, if your CI comes up red with some unrelated test failing, you can mark the test as flakey in the UI and CI will allow your code to merge and a Jira ticket w…
That sounds like it works, but shouldn't flaky tests be detected by frequent automatic test suite runs?
Re: Software testing, and why I'm unhappy about it
#9Doctor, it hurts when I punch myself in the head!
If testing that way is painful (and it is), then work with people to remove the pain. Tests are supposed to help developers, not constrain or punish them.
Put tests in the same repo as the SUT. Do more testing closer to the code (more service and component tests) and do less end-to-end testing. Ban "flakey" tests - they burn engineering time for questionable payoff.
Test failures can be thought of as "things developers should investigate." Make sure the tests are focused on telling you about those things as fast as possible.
Also, take the human out of the "wait for green, then submit PR" steps. Open a PR but don't alert everyone else about it until you run green, maybe?
Re: Software testing, and why I'm unhappy about it
#10The author's problem is pretty simple: the test repo is required for pre-merge tests to pass, but it can be updated independently, without having pre-merge tests pass.
And the answer is pretty simple: pin the specific test repo version! Use lockfiles, or git submodules, or put "cd tests && git checkout 3e524575cc61" in your CI config file _and keep it in the same repo as source code_ (that part is very important!).
This solves all of author problems:
> new test case is added to the conformance test suite, but that test happens to fail. Suddenly nobody can submit any changes anymore.
Conformance test suite is pinned, so new test is not used. A separate PR has to update conformance test suite version/revision, and it must go through regular driver PR process and therefore must pass. Practically, this is a PR with 2 changes: update pin and disable new test.
> are you going to remember to update that exclusion list?
That's why you use "expect fail" list (not exclusion) and keep it in driver's dir. Ad you submit your PR you might see a failure saying: "congrats, test X which was expect-fail is now passing! Please remove it from the list". You'll need to make one more PR revision but then you get working tests.
> allowing tests to be marked as "expected to fail". But they typically also assume that the TB can be changed in lockstep with the SUT and fall on their face when that isn't the case.
And if your TB cannot be changed in lockstep with SUT, you are going to have truly miserable time. You cannot even reproduce the problems of the past! So make sure your kernel is known or at least recorded, repos are pinned. Ideally the whole machine image, with packages and all is archived somehow -- maybe via docker or raw disk image or some sort of ostree system.
> Problem #2 is that good test coverage means that tests take a very long time to run.
The described system sounds very nice, and I would love to have something like this. I suspect it will be non-trivial to get working, however. But meanwhile, there is a manual solution: have more than one test suite. "Pre-merge" tests run before each merge and contain small subset of testing. A bigger "continuous" test suite (if you use physical machines) or "every X hours" (if you use some sort of auto-scaling cloud) will run a bigger set of tests, and can be triggered manually on PRs if a developer suspects the PR is especially risky.
You can even have multiple levels (pre-merge, once per hour, 4 times per day) but this is often more trouble than it worth.
And of course it is absolutely critical to have reproducible tests first -- if you come up to work and find a bunch of continuous failures, you want to be able to re-run with extra debugging or bisect what happened.