A survey of testing techniques we've found useful
1–10 of 23 posts
Re: A survey of testing techniques we've found useful
#2I think this is a fairly good summary of the most important testing styles and where / when to (not) use them.
One more category of tests I would add are meta tests (like mutation tests). These are tests which test the tests, seeing if they would actually catch any errors / bugs or just report everything to be alright always.
Re: A survey of testing techniques we've found useful
#3"If something is difficult to unit test, refactor until it's easy."
This is often a good idea but if you only need the flexibility to enable unit testing then it may make your system more complex than it needs to be. Only introduce indirection where it's really needed. See also "test induced design damage" and "write tests, not too many, mostly integration".
Re: A survey of testing techniques we've found useful
#4I’m a complete newbie when it comes to writing tests. But I know that SQLite is tested to hell and back, and I believe Richard Hipp (the creator) has said he spends more time and lines of code on the testing suite, than on the SQLite code itself. I hope he shares some of his insights some day.
Re: A survey of testing techniques we've found useful
#5I’m a complete newbie when it comes to writing tests. But I know that SQLite is tested to hell and back, and I believe Richard Hipp (the creator) has said he spends more time and lines of code on the testing suite, than on the SQLite code itself. I hope he shares some of his insights some day.
> I hope he shares some of his insights some day.
You didn't search much: https://sqlite.org/testing.html
Re: A survey of testing techniques we've found useful
#6"If something is difficult to unit test, refactor until it's easy." This is often a good idea but if you only need the flexibility to enable unit testing then it may make your system more complex than it needs to be. Only introduce indirection where it's really needed. See also "test induced design damage" and "write tests, not too many, mostly integration".
Only if you want to minimize the total complexity, which is almost never a good idea. What is more important, is the minimum amount of complexity needed to be understood in order to make a change. If you are able to test a small subset of components in isolation, you can also understand them in isolation, and modify them without the need to understand the whole system. I'd rather read 20% of 110% code than 100% of 100%.
The advice to write mostly integration tests is a terrible one. Particularly when they test integrating of everything. When such tests catch bugs, they don't tell where the problem happened. They also take long time to execute.
Re: A survey of testing techniques we've found useful
#7One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accuracy. At best you can declare a baseline, eg that your algo should be say 90% accurate (if you implemented it properly based on literature results) and if you don't get that, then error. In that case you can use a numerical metric, but other applications you might care about the result being aesthetically pleasing (eg a video ISP where you do colour correction on the stream coming from a low level camera).
Or hardware where the advice is usually to mock the device under test. But if you don't own the hardware the most you can do is try and emulate it, and maybe check that your simulated state machine works. In my experience its easier to run with hardware connected and just skip those tests otherwise. There are also extremely subtle bugs that can crop up with hardware interfaces like needing to insert delays into code (eg when sending serial) that will otherwise fail in the real world.
OpenCV has some interesting approaches to this, for example testing storing a video in a certain format, inserting a frame with a known shape (like a circle), then reading back the video and checking that the shape can be detected.
Re: A survey of testing techniques we've found useful
#8When writing code, we often think "according to the design of our system, this condition must be true at this point of execution.” Examples are:
1. The argument x must not be 0.
2. The variable x must smaller than the variable y.
3. The list foo must be non-empty.
4. The variable x should have value 'Success' if it had value 'Try' in the beginning of the function call.
These 'invariants', or assertions, can be extremely useful for testing the correctness of the code. Put simply, if an invariant is violated (during unit test, integration tests, or system tests), it indicates that either the design or the implementation is wrong. An article on testing methodology would be more appealing if it had some discussion on exploiting invariants/assertions.
Re: A survey of testing techniques we've found useful
#9When writing code, we often think "according to the design of our system, this condition must be true at this point of execution.” Examples are: 1. The argument x must not be 0. 2. The variable x must smaller than the variable y. 3. The list foo must be non-empty. 4. The variable x should have value 'Success' if it had value 'Try' in the beginning of the function call. These 'invariants', or assertions, can be extrem…
I believe "Design by Contract"[0], or "DbC", is the concept you are describing. In the Wiki page Notes and External Links sections there are some good resources IMHO.
Re: A survey of testing techniques we've found useful
#10One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accu…
I only mentioned it briefly at the end of the post, but metamorphic testing is a very interesting technique that addresses exactly this [0].
The basic idea is to start with some known-good inputs and outputs, and then generate ways to modify the input that should not change the output.