Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using `make` and `git diff` for a simple and powerful test harness

chrismorgan.info

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Using `make` and `git diff` for a simple and powerful test harness

#21

Post by drothlis »

This technique is also known as characterisation testing, golden-master testing, snapshot testing, and probably other names too. I recommend looking into https://approvaltests.com/ (already mentioned by another commenter).

I use characterisation testing all the time, in a perhaps unusual application: Checking the behaviour of "Page Objects" (classes used to model the application-under-test in GUI-level automated testing) when the application-under-test has changed. That's right: Automated [characterisation] tests for my automated [GUI] tests! It makes GUI testing oh so much easier to maintain. I wrote it up here: https://david.rothlis.net/pageobjects/characterisation-tests

Re: Using `make` and `git diff` for a simple and powerful test harness

#22

Post by heinrichhartman »

Note that GNU CoreUtils also use Make + shell-scripts + diff for their test harness: https://github.com/coreutils/coreutils/tree/master/tests

Its not as slick or concise as the solution proposed here, but it shows that the approach is viable for medium-large projects.

Re: Using `make` and `git diff` for a simple and powerful test harness

#23

Post by code-faster »

I've used a similar technique to test, generating an expected output, actual output and then diff them.

One trick I found helpful was using JSON to serialize test results instead of unstructured plain text.

Test results stored as JSON are much easier parse and therefore process. You can quickly whip up programs that verify the tests satisfy invariants, diff the tests and filter out expected test changes from unexpected test changes.

Re: Using `make` and `git diff` for a simple and powerful test harness

#24

Post by drothlis »

drothlis wrote: post #21

This technique is also known as characterisation testing, golden-master testing, snapshot testing, and probably other names too. I recommend looking into https://approvaltests.com/ (already mentioned by another commenter). I use characterisation testing all the time, in a perhaps unusual application: Checking the behaviour of "Page Objects" (classes used to model the application-under-test in GUI-level automated test…

Another thought related to characterisation testing, this time from Jeremias Roessler's talk at QAFest Ukraine 2017 (https://www.youtube.com/watch?v=f4PT_u8hjhU): Traditional automated tests (with asserts) are "blacklisting" the changes that aren't allowed, whereas a characterisation test will catch any change in behaviour, and instead you have to "whitelist" the changes that are allowed (by providing regexes or some kind of pattern to match the dynamic output that is allowed to change).

Re: Using `make` and `git diff` for a simple and powerful test harness

#26

Post by pgt »

drothlis wrote: post #21

This technique is also known as characterisation testing, golden-master testing, snapshot testing, and probably other names too. I recommend looking into https://approvaltests.com/ (already mentioned by another commenter). I use characterisation testing all the time, in a perhaps unusual application: Checking the behaviour of "Page Objects" (classes used to model the application-under-test in GUI-level automated test…

What I like about this idea of Approval Testing is that I don't have to write too many tests if I know my system is working, as long as I am notified when the behaviour of my system has changed (usually in some unexpected way).

This would require auto-mocking of all subsystems to prevent side-effecting functions from sending email & SMSes.

Re: Using `make` and `git diff` for a simple and powerful test harness

#28

Post by qznc »

code-faster wrote: post #23

I've used a similar technique to test, generating an expected output, actual output and then diff them. One trick I found helpful was using JSON to serialize test results instead of unstructured plain text. Test results stored as JSON are much easier parse and therefore process. You can quickly whip up programs that verify the tests satisfy invariants, diff the tests and filter out expected test changes from unexpect…

When I did it, we skipped the Make part and reinvented it in Python: https://github.com/libfirm/sisyphus

This aspect of expected and unexpected test changes is even more important than the diff part in my opinion. It allows you add failing tests immediately once you get the bug report and you notice if you fix something accidentally.

Re: Using `make` and `git diff` for a simple and powerful test harness

#29

Post by jgrahamc »

chrismorgan wrote: post #20
jgrahamc wrote: post #19

Fun. The rwildcard function here rwildcard = $(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2) $(filter $2,$d)) is very similar to one I wrote in 2011 ( https://blog.jgc.org/2011/07/gnu-make-recursive-wildcard-fun... ): rwildcard=$(foreach d,$(wildcard $1*),$(call rwildcard,$d/,$2) $(filter $(subst *,%,$2),$d)) The only difference is that I wanted to be able to use * in the pattern rather than % (hence the subst).…

I believe it is a direct descendant of yours, which I’ve used from time to time over the years. I just didn’t feel the need to support *, % was enough for me. Thanks for all the stuff you’ve written about Make: I’ve enjoyed reading quite a lot of it!

Glad these things help people. The * for wildcard was a bit of syntatic sugar to make $(call rwildcard) closer to $(wildcard).

For those who don't know, here's a list of stuff I've written over the years.

https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-art...

And the hard copy version: https://nostarch.com/gnumake

PS I really like the layout of your blog.

Re: Using `make` and `git diff` for a simple and powerful test harness

#30

Post by tom_mellior »

qznc wrote: post #28
code-faster wrote: post #23

I've used a similar technique to test, generating an expected output, actual output and then diff them. One trick I found helpful was using JSON to serialize test results instead of unstructured plain text. Test results stored as JSON are much easier parse and therefore process. You can quickly whip up programs that verify the tests satisfy invariants, diff the tests and filter out expected test changes from unexpect…

When I did it, we skipped the Make part and reinvented it in Python: https://github.com/libfirm/sisyphus This aspect of expected and unexpected test changes is even more important than the diff part in my opinion. It allows you add failing tests immediately once you get the bug report and you notice if you fix something accidentally.

The Readme of that project could do with a few examples of what tests and successful/unsuccessful output look like. I found the examples folder and still can't visualize what it might be like.

I've been using cram (also written in Python) for a private project and been mostly happy with it: https://bitheap.org/cram/

Post reply on HN