31–40 of 115 posts
It would do a better job with the mental game. You may not be able to conclude that code is adequately tested just by knowing that the tests exercise it. But you can be pretty confident that code is not being tested if it eludes all of the tests.
Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…
Right on point. High CC% doesn't tell application is well tested but is an indicator of what isn't covered
Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…
Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.
This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program.
Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives 100% code coverage but it covers at the very best only 25% of the code behaviors.
I find the opposite to be the case. I'm very comfortable tossing away a bunch of code with great test coverage. You can always cherry-pick it back later and know that it works.
(I'm not 100% purist, but certainly I'd say 80-90+% and not 66%).
The important thing is you should be so fast at writing testable code that getting to 80% is a like going for a light jog. If it feels like a slog, you've identified weaknesses in your craftmanship that you should practice.
Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…
#1 source will be the code that doesn't appear at all (because the test build system didn't even include it) #2 will be all those red lines.
There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end test…
Of course there are times where you need these weakly typed languages with lots of coverage. In a sense that is what the Web is, and the browser vendors use millions of websites for their coverage.
I'm a strident adherent to having 100% code coverage, but this mostly works because my history in infrastructure and a desire to sleep well. The key idea that I have found that if you can't synthetically get your process into a specific state, then life is going to be hard. The problem with a lot of people is that they view testing as a burden rather than a criticism of the code they are testing. If you can't test yo…
You must be trolling.
I don't think this is trolling. Maybe I would have phrased it in a nicer way, but I agree with their math.
Now I don't think you need to get to 100%, but if you can't easily see how you could get to 100%, that's a problem.
Earlier quoted context omitted.
I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org
whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well. For Python we use mutmut [1]. [1] - https://pypi.org/project/mutmut/
I think it's pretty important that we who know about MT and have used it keep talking about it. It's a pretty great tool that more people should have in their tool belt.
There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end test…
That's interesting because I think the exact opposite. For dynamically typed languages I think it's more valuable to have end-to-end tests to make sure that the whole pipeline works correctly. I want to validate that all the call sites for function F are passing an int, as intended, as opposed to a string. For statically typed languages, I already know that all call sites for F are passing an int. So I want to unit t…
Earlier quoted context omitted.
Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.
test coverage is calculated as % of the locations seen by profiler during the test runs, right ? This metric completely ignores the fact that every independent conditional doubles the space of the behaviors of the program. Assuming we have three conditions and three sequential groups of basic blocks chosen corresponding to true (X) and false (~X) values of these conditions, testing A+B+C and ~A+~B+~C code paths gives…
> Full path coverage, of the type described above, is usually impractical or impossible. ... a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).[14] Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage."