A survey of testing techniques we've found useful • Hacker News Board

Re: A survey of testing techniques we've found useful

#21

Post by ptsneves » Sun, Apr 12, 2020, 8:31 AM UTC

dbcurtis wrote: post #15

Ididntdothis wrote: post #11

Earlier quoted context omitted.

Testing with hardware is hard. You can emulate to some degree but that just helps making sure your tests are written correctly. In the end you have to run tests against the real thing. I deal with complex UIs that interact with hardware. If you are smart you can split things up so they are easier to test in isolation but the whole system has a ton of potential interactions that are hard to write test cases for. The O…

> Testing with hardware is hard. Yup. I work in robotics. I try to isolate the actual hardware interaction layer so that for testing you can mock the driver and hardware in one piece. Of course that does not test the driver. With any luck, the driver is pretty stable once it works, though. And the driver+hardware piece can have it's own (physical) test bench so that at least manual testing is, well maybe not efficien…

Nice write up. Just wanted to add that the problem of using simulators or mock is that now you have one extra code base to maintain totalling: the code, the test, and the mock. For mock drivers this can be a quite big task. In the end I just preferred to run it in real hardware as much as possible and go for unit tests. This from a person that generally does not like unit tests, but there is not a very cheap way of going with simulations sometimes.

Re: A survey of testing techniques we've found useful

#22

Post by ptsneves » Sun, Apr 12, 2020, 8:35 AM UTC

choeger wrote: post #20

Very nice overview. I very much agree that all these techniques are very useful. But one thing still bothers me. How does one test with a notion of time? How do you test a cron or a calendar with alarm function?

Yeah, those are the hard ones. Worse, time is the essence of a good amount of issues from performance, to startup stabilization to the deadliest of all errors: the race condition.

Re: A survey of testing techniques we've found useful

#23

Post by joshvm » Sun, Apr 12, 2020, 5:53 PM UTC

Ididntdothis wrote: post #11

joshvm wrote: post #7

One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accu…

Testing with hardware is hard. You can emulate to some degree but that just helps making sure your tests are written correctly. In the end you have to run tests against the real thing. I deal with complex UIs that interact with hardware. If you are smart you can split things up so they are easier to test in isolation but the whole system has a ton of potential interactions that are hard to write test cases for. The O…

Emulation works well if you have firmware and an emulator. For example, the Ardupilot autopilot software has both hardware-in-the-loop and software-in-the-loop packages which use the actual firmware. It runs off an STM32 emulator (I think) which is well defined. As you say, if you don't have that firmware, your emulator is only as good as your reverse engineering is.

When I'm testing thermal cameras there are a sequence of things I can check to ensure that the test worked: was the command sent without errors? Did I get an error back from the camera (e.g. CRC failure)? Does the state of the camera change as I expect it to? If all of those things are correct then the likelihood is that the command sent OK. Of course for states you should check various permutations (e.g. shutter open and shutter closed) to make sure that you don't have a bug in your state reading code :)

Here's a stereo matching example from OpenCV. This is a case when you do have the correct answer, but you don't expect to equal it, and your tolerance to accuracy varies with algorithm:

https://github.com/opencv/opencv/blob/055645080161c6af6083b6...