Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A survey of testing techniques we've found useful

vector.dev

Post reply on HN
Print view View original

11–20 of 23 posts

Re: A survey of testing techniques we've found useful

#11

Post by Ididntdothis »

joshvm wrote: post #7

One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accu…

Testing with hardware is hard. You can emulate to some degree but that just helps making sure your tests are written correctly. In the end you have to run tests against the real thing. I deal with complex UIs that interact with hardware. If you are smart you can split things up so they are easier to test in isolation but the whole system has a ton of potential interactions that are hard to write test cases for.

The OpenCV example is a pretty easy one. You have clear inputs with clearly defined outputs. The only thing you have to do is to create sample data.

Re: A survey of testing techniques we've found useful

#12

Post by carlosf »

joshvm wrote: post #7

One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accu…

> How do you check your code works?

When I think about my projects "working", I always try to answer the following questions:

1) Is my code doing what I believe it should be doing? That question always have objective answers and is the subject of software engineering testing.

2) Is my solution solving my problem efficiently? Often that's a domain specific question and different domains have different ways of doing quality assurance, there's no silver bullet.

Re: A survey of testing techniques we've found useful

#13

Post by cuchoi »

epdlxjmonad wrote: post #8

When writing code, we often think "according to the design of our system, this condition must be true at this point of execution.” Examples are: 1. The argument x must not be 0. 2. The variable x must smaller than the variable y. 3. The list foo must be non-empty. 4. The variable x should have value 'Success' if it had value 'Try' in the beginning of the function call. These 'invariants', or assertions, can be extrem…

Aren't these examples of unit tests?

Re: A survey of testing techniques we've found useful

#14

Post by epdlxjmonad »

AdieuToLogic wrote: post #9
epdlxjmonad wrote: post #8

When writing code, we often think "according to the design of our system, this condition must be true at this point of execution.” Examples are: 1. The argument x must not be 0. 2. The variable x must smaller than the variable y. 3. The list foo must be non-empty. 4. The variable x should have value 'Success' if it had value 'Try' in the beginning of the function call. These 'invariants', or assertions, can be extrem…

I believe "Design by Contract"[0], or "DbC", is the concept you are describing. In the Wiki page Notes and External Links sections there are some good resources IMHO. 0 - https://en.wikipedia.org/wiki/Design_by_contract

Yes, closely related, but invariants can appear anywhere in the code (like loop invariants), and are less restrictive than pre-conditions and post-conditions which must appear in the beginning and end of methods. So, invariants are more about testing than design.

Arguably, invariants are especially powerful in testing distributed systems:

0 - https://www.datamonad.com/post/2020-02-19-testing-mr3/

Re: A survey of testing techniques we've found useful

#15

Post by dbcurtis »

Ididntdothis wrote: post #11
joshvm wrote: post #7

One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accu…

Testing with hardware is hard. You can emulate to some degree but that just helps making sure your tests are written correctly. In the end you have to run tests against the real thing. I deal with complex UIs that interact with hardware. If you are smart you can split things up so they are easier to test in isolation but the whole system has a ton of potential interactions that are hard to write test cases for. The O…

> Testing with hardware is hard.

Yup. I work in robotics.

I try to isolate the actual hardware interaction layer so that for testing you can mock the driver and hardware in one piece. Of course that does not test the driver. With any luck, the driver is pretty stable once it works, though. And the driver+hardware piece can have it's own (physical) test bench so that at least manual testing is, well maybe not efficient, but at least not painful.

Simulators are great but not always available. Or are too much work to get going.

One configuration often used for robots is the "boneless chicken". Take a bench, and bolt all the guts down to it in a configuration where they are easy to probe. Put the wheel motors someplace safe, with a synthetic load like a pony brake. Of course you can't test the nav stack that way. (I once interviewed a firmware engineer who was coming off of the Juicero shutdown -- say what you want about Juicero, but from the sounds of it their boneless chicken was outstanding, even integrated into the CI automation pipeline. Of course, they didn't have the nav problem).

Speaking of nav, I once saw a warehouse robot company's nav PR test micro-warehouse. Not the full test warehouse, just a 500 square foot or so area dedicated to testing nav PR's. It was integrated with CI automation. I could tell from the accumulated tire marks on the floor that they had nav pretty much nailed.

I have done several robot-to-elevator interfaces (probably more than anyone else). In the end, final testing always required something akin to a few midnight to 4 AM test blocks on the real elevator. And then of course as you point out:

> the whole system has a ton of potential interactions that are hard to write test cases for.

They often don't show up until the system is under load.

Re: A survey of testing techniques we've found useful

#16

Post by binarylogicben »

idoubtit wrote: post #5
RaoulP wrote: post #4

I’m a complete newbie when it comes to writing tests. But I know that SQLite is tested to hell and back, and I believe Richard Hipp (the creator) has said he spends more time and lines of code on the testing suite, than on the SQLite code itself. I hope he shares some of his insights some day.

> I hope he shares some of his insights some day. You didn't search much: https://sqlite.org/testing.html

I love this page! We (Vector) hope to post something similar in the future.

Re: A survey of testing techniques we've found useful

#17

Post by MaxBarraclough »

cuchoi wrote: post #13
epdlxjmonad wrote: post #8

When writing code, we often think "according to the design of our system, this condition must be true at this point of execution.” Examples are: 1. The argument x must not be 0. 2. The variable x must smaller than the variable y. 3. The list foo must be non-empty. 4. The variable x should have value 'Success' if it had value 'Try' in the beginning of the function call. These 'invariants', or assertions, can be extrem…

Aren't these examples of unit tests?

No, as they're runtime asserts baked into the code itself, rather than existing elsewhere.

Re: A survey of testing techniques we've found useful

#18

Post by taeric »

pkolaczk wrote: post #6
discreteevent wrote: post #3

"If something is difficult to unit test, refactor until it's easy." This is often a good idea but if you only need the flexibility to enable unit testing then it may make your system more complex than it needs to be. Only introduce indirection where it's really needed. See also "test induced design damage" and "write tests, not too many, mostly integration".

Only if you want to minimize the total complexity, which is almost never a good idea. What is more important, is the minimum amount of complexity needed to be understood in order to make a change. If you are able to test a small subset of components in isolation, you can also understand them in isolation, and modify them without the need to understand the whole system. I'd rather read 20% of 110% code than 100% of 10…

The problem I have been exposed to is unit tests that have locked in what should be just an implementation detail. Can be fine, if it is an important detail. That said, your tests should not have to import all of the same pieces that your coffee does. I prefer, actually, to use alternatives in the tests, where possible.

For example, if you're code zips something, your test could use many zip engines to verify.

Re: A survey of testing techniques we've found useful

#19

Post by kqr »

joshvm wrote: post #7

One thing that is rarely discussed (I think?) is how to test things which don't have a correct answer. It's not just "refactor until you can test" it's output that may be subjective. For example, suppose you write some code to do some image processing like a stereo matcher. How do you check your code works? Usually you have some ground truth which you can compare, but it's difficult because you'll never get 100% accu…

Tell me about it! I work on a search engine powered by relatively basic machine learning of user behaviour. We probably achieve the most relevant results in the world for our customers. That's not the hard part.

The hard part is tightening our development feedback cycle. Since we outperform all competitors, we don't have an oracle to test against. We can automate testing with a small sample of input-output pairs, but the brunt of the work is still done by humans trained and paid to judge the quality of the results. It's an awful position to be in.

I have started looking for better ways of doing it, and the most promising I've found so far is metamorphic testing, mentioned in another comment.

Property testing only takes you a short bit here, as far as I've been able to figure out.

(I have also glanced at the techniques used in bioinformatics, since those guys are good at comparing sequences, but that's more specific to our case and not a general solution.)

Post reply on HN