121–130 of 178 posts
I'm sorry what? Of course if you only run a random subset of tests, it won't matter whether the test suite achieves 100% line coverage. What a garbage study.
Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test c…
It is very unpopular view. I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented. This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client. Since the outside interface of the…
The one exception is for utility functions. I advocate TDD for utility functions, with heavy testing of standard and edge cases, to ensure that the functions conform to spec and any future tweaks / refactors of the functions continue to conform to spec.
Earlier quoted context omitted.
Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…
I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.
Though I'd probably write:
void fn(int a, int b)
{
printf(a == b ? "equal" : "not");
}"Next, they selected random subsets of the project's original test suite of varying sizes, ran each against the buggy mutants, and counted how often the subsetted test suite caught the bug." I'm sorry what? Of course if you only run a random subset of tests, it won't matter whether the test suite achieves 100% line coverage. What a garbage study.
They are stating that having X% code coverage does not imply that you will be able to catch X% of bugs (at least, of the trivial type they introduced here) so you shouldn't use code coverage as a proxy for how good your test suite is.
This may be obvious to you, but I've definitely seen teams fall into the trap of using code coverage as a primary measure of how good testing is.
I've been asked many times what is the right code coverage percentage to aim for. I've also asked this many times in interviews, to tease out a discussion. My answer is: you need full coverage. I then continue to explain that I didn't say 100%. Although I believe that the average Java microservice (which is the general sphere that I move around) can easily achieve >98% coverage. Easily. But "full coverage" means that…
1. It makes the code too inflexible to refactor, leaving the system too rigid to grow.
2. It makes people frustrated on how to write tests in this project.
3. It makes people spend more time on writing tests than implementation.
So some quick tips:
1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.)
2. Be simple, clear, and consistent about how to write test on the project. If it's layered architecture, be clear about which layers need to be tested. And be minimal, two layers of testing is usually more than enough. Make sure the testing rules is enforced in the code base in the beginning, and then it will be downhills since people are mostly copying.
3. People always talk about the Testing Pyramid: the idea is coarse-grained tests are valuable but too costly while the fine-grained tests are cheap, so it's all trade-off and the optimal is to make them like a pyramid. But in reality, certain layers of tests could be drastically optimized. The idea is maximize the granularity on behaviors covered, while minimize the infrastructure involved (e.g. carefully designed monolith + hexagonal architecture), or make the infrastructure fast and parallelizable (e.g. SQL sandboxing). The end result is a certain layer could be a better trade-off than others, so concentrate tests there instead of writing tests repeatedly.
Earlier quoted context omitted.
Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…
I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.
Earlier quoted context omitted.
Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…
I wouldn't even write this way because the else confuses what's happening. void fn(int a, int b) { string ret = "not"; if (a == b) ret = "equal"; printf(ret); } *I understand, this is not a good optimization in every language.
Though I would also accept:
void fn(int const a, int const b)
{
printf(a == b ? "equal" : "not");
}
fn theFn(a: isize, b: isize)
{
let ret;
if a == b {
ret = "equal";
} else {
ret = "not";
}
println!("{}", ret);
}
The typical CRUD apps out here in the contracting world - many in serious disrepair after a few different offshore or junior teams have worked on them - have been designed using way too many microservices and databases, then duct taped together with message queues and "Cloud" infrastructure like SNS or Lambda.
Unit tests are pointless since 90% of your code does not fit the typical use case they'll teach you about in Bootcamp or in some TDD book - it's not like you're implementing some linked list implementation where you can write a nice, self-contained test suite that goes through a few valid inputs, edge cases, and proves things work.
Instead, you're just assembling dozens of frameworks and managed services and 3rd party libraries, and wrapping it all up with a UI on the front.
Integration tests used to be helpful. But nowadays, you'd need to recreate some huge Docker environment for each test, complete with several pre-loaded databases, half a dozen supporting microservices, and ensure you can connect to a complete test environment in the cloud for your managed infrastructure.
Or you can do what's becoming increasingly common in the Java / Spring world where you have this elaborate system of mocks. Eventually, your mock classes will become more complex and elaborate than the original class itself. You'll spend 50% of your time trying to keep this integration environment working, with all the race conditions it entails as databases, queues, services sometimes start "out of order", and it won't even provide any benefit, because your mocks don't behave at all like the non-mocks do outside of tests.
And don't get me started on front-end testing these days. What I see most amateur developers doing is just instantiating their little React components in some pseudo-browser environment and then testing that, yes, some internal class used by the component library is, in fact, also applied on their test component. Which means they're not testing anything relevant at all. And it also means all the tests break when we move to the next version of the component library, and all those internal classes are renamed or deprecated.