251–260 of 346 posts
Earlier quoted context omitted.
Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)
> testing should be built into the language itself I wonder what that would look like...
Earlier quoted context omitted.
I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…
> even private ones. You don't test private API, that's why they are declared private, it makes absolutely no sense to test private members. Now the use of the private member might lead to a different code path : it has to be tested. Unit Testing : test one unit ( a class for instance ) in isolation, which means all collaborators (the classes the tested class depends on) have to be stubbed or mocked. Functional Testi…
Don't agree. I think this is something that people tell themselves to justify the anti-pattern that is private members being nearly impossible to test directly in a variety of languages, but it seems like nonsense. I've never seen a real case for excluding private methods as a testable unit.
1) 100% path coverage is not even close to exhaustively checking the full set of states and state transitions of any usefully large program.
2) If, furthermore, you have concurrency, the possible interleavings of thread execution blow up the already-huge number of cases from 1) to the point where the latter look tiny in comparison.
3) From 1) and 2), it is completely infeasible to exhaustively test a system of any significant size.
The corollary of 3) is that you cannot avoid being selective about what you test for, so the question becomes, do you want that decision to be an informed one, or will you allow it to be decided by default, as a consequence of your choice to aim for a specific percentage of path coverage?
For example, there are likely to many things that could be unit-tested for, but which could be ruled out as possibilities by tests at a higher level of abstraction. In that case, time spent on the unit tests could probably be better spent elsewhere, especially if (as with some examples from the article) a bug is not likely.
100% path coverage is one of those measures that are superficially attractive for their apparent objectivity and relative ease of measuring, but which don't actually tell you as much as they seem to. Additionally, in this case, the 100% part could be mistaken for a meaningful guarantee of something worthwhile.
Earlier quoted context omitted.
If a function takes a single 32 bit argument you can run it on every possible value.
… this reply does nothing to answer the parent's question. It's obvious that you can run it on all possible values, but that doesn't answer the question of the return values. (unless you only want to verify "doesn't crash" or something similarly basic)
The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…
This is obviously wrong.
> Tests will be poorly written, pointless, and give an overall false sense of security to the next sap who breaths a sigh of relief when "nothing is broken". Of course, that house of cards will come down the first time something is in fact broken.
That's a generalization that does not apply everywhere.
Earlier quoted context omitted.
I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…
How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.
FLOAT_EQ(sqrt(x) * sqrt(x), x) && sqrt(x)>=0.0
A lot of people here seem to have strong opinions against 100% coverage, so I'll risk their ire with my strong opinion in favor. If you have, say, 95% coverage -- and most corporate dev orgs would be thrilled with that number -- and then you commit some new code (with tests) and are still at 95%, you don't know anything about your new code's coverage until you dig into the coverage report. Because your changes could…
The problem with the "100%" goal is that it provides a false sense of safety. There can still be plenty of bugs in code that has "100%" test coverage. The 100% are an illusion. To truly get to 100%, it does not suffice that every line of code is executed, you must also test all valid inputs - which is utopic.
If you know what's going on -- and hopefully you do if you're making decisions like whether to pursue a 100% coverage goal -- then your point is (I hope) self-evident.
It's also why code review is necessary and not just the coverage metric. 100% coverage is not "100% of all things that could happen," it's "100% of all code execution paths" and (hopefully) a sample of all known types of valid and invalid inputs.
Anyway, so far I've found that most developers respond pretty well to the 100% idea if you explain its utility and what it does and doesn't get you. From a ridiculously small sample, granted.
The utopian risk, I think, is much more around the hope that people high up the food chain in your organization actually understand this stuff. Consider how many VPs of Engineering did a year or two of engineering work (before unit testing was in vogue) then got that MBA and have been busy taking meetings ever since.
/me raises hand on the pro-testing side I've been programming for a living since 1996, and only recently started to do TDD in the normal sense of writing unit tests before writing code. I've found to it to be an enormous help with keeping my code simple - the tests or the mocking getting difficult is a great indicator that my code can be simplified or generalised somehow I argued for functional instead of unit testin…
I agree that TDD and unit tests can help you when writing code. But once your code has been written, are all those unit tests still useful? Or do they get in your way when you do large scale refactoring? I have seen quite a few codebases with loads of unit tests that might once have been useful, but now were just slowing down development. Many of them should have been thrown away and replaced by a couple of functiona…
This is a REALLY common developer attribute that drives me nuts - people don't want to throw away stuff that they put a lot of effort into.
Earlier quoted context omitted.
I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…
How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.
Some times I have, for example, matlab code that I 'know' is right and want to verify that my C/Python/Julia... code does the same thing. So then I just call the matlab code, call the new code and check if they're the same.
Or I have a slow exact method and want to see if my faster approximation always gives an answer 'close enough' to the correct answer.
I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes = for example), and then runs your tests. If one test fails, your tests covered that statement. It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim. The other option is t…