How to Write Unit Tests for Logging • Hacker News Board

Re: How to Write Unit Tests for Logging

#11

Post by rwbhn » Wed, Nov 11, 2020, 12:38 PM UTC

ThePadawan wrote: post #7

sgt101 wrote: post #3

Let's write unit tests for unit tests!

To be fair, I worked at a company that had CI for running unit tests so nothing could be merged into remote master until the build was green. The build asserted that the number of failing unit tests was 0. It turns out that "breaking the unit test library completely such that 0 tests run instead of all thousands of them" did not break the build. When someone noticed a week later, that was not fun to clean up.

Recently added an assertion that the # of tests succeeding was > 0 to cover a similar issue.

Re: How to Write Unit Tests for Logging

#12

Post by henrik_w » Wed, Nov 11, 2020, 12:49 PM UTC

I usually don't test the logging. However, sometimes I get tests for the logging messages "for free". That happens when I break out some logic in its own function, and I want logging on what happened in that function (definitely not all the time, but is sometimes useful). Then the function can return its result, and a list of logging messages to output. When asserting the result, I can also assert on the logging messages.

It's what I describe in "Returning a message list" in this post: https://henrikwarne.com/2020/07/23/good-logging/

Re: How to Write Unit Tests for Logging

#13

Post by luxurytent » Wed, Nov 11, 2020, 1:06 PM UTC

sethammons wrote: post #10

In Go, I test my log output easily: my logger instance’s output is set to a buffer, call the function under test, assert log contents. If another system depends on it, it gets tested. Our acceptance level tests operate largely on logs.

I've mostly spent my career in Python/Javascript and recently jumped back into Go. How a Go programmer can use an interface to effectively mock another object feels very natural and core to the language, I like it.

(I'm probably using incorrect terminology here, so happy if someone corrects me)

Re: How to Write Unit Tests for Logging

#14

Post by geewee » Wed, Nov 11, 2020, 1:15 PM UTC

valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

Completely agree with this. I've wanted to test type A logs, and often heard people saying that "you should never test logs" No - you should never test most types of logs - sometimes logs, particularly in error situations, are part of the output of a function.

Re: How to Write Unit Tests for Logging

#15

Post by pydry » Wed, Nov 11, 2020, 1:23 PM UTC

u801e wrote: post #8

It seems that the boundaries of what a unit test should cover have never been clearly defined. In my opinion, unit testing should only cover logic that does not have side effects (like writing to disk or communicating over a network). If you want to test external side effects, you should write functional or integration tests for them.

I wish more people believed this. The number of lines of code I've seen written of pointless unit tests is staggering.

If your code is mostly passing data around and managing side effects a unit test is more like a parasite. It will couple to your code and fail noisily when refactoring but not provide any actual useful feedback.

Re: How to Write Unit Tests for Logging

#16

Post by tlarkworthy » Wed, Nov 11, 2020, 1:29 PM UTC

You can use log and monitoring signals to whitebox test certain implementation details to an otherwise stateless interface.

I would say it's not great practice, but, it can remove a lot of contortions to be able to make assertions about the execution path taken when doing a unit test.

If I have a tricky thing to test. I ask, if I added the required operational monitoring (e.g. queue sizes)? Would I suddenly be able to test this easier? Would it make the test more self explanatory. If the answer is yes I might add the monitoring and then do the unit testing. If the answer is no I have to refactor the implementation for testability.

conclusion: exposing logs and especially monitoring counters to unit tests can actually super charge your testing and leave you net positive

Re: How to Write Unit Tests for Logging

#17

Post by the_af » Wed, Nov 11, 2020, 2:51 PM UTC

sgt101 wrote: post #3

Let's write unit tests for unit tests!

> Let's write unit tests for unit tests!

Tests for tests exist. One such technique is called "mutation testing" [1].

I don't know of any job that actually uses them, but they make sense to me: the tests need checks too, otherwise how do you know they are adding value? It's too easy to write worthless tests, see them pass, and get a false sense of security. Techniques such as mutation testing check how sensitive your tests are to actual bugs (or program changes): do they detect them? If not, they aren't useful.

[1] https://en.wikipedia.org/wiki/Mutation_testing

Re: How to Write Unit Tests for Logging

#18

Post by the_af » Wed, Nov 11, 2020, 2:53 PM UTC

ThePadawan wrote: post #7

sgt101 wrote: post #3

Let's write unit tests for unit tests!

To be fair, I worked at a company that had CI for running unit tests so nothing could be merged into remote master until the build was green. The build asserted that the number of failing unit tests was 0. It turns out that "breaking the unit test library completely such that 0 tests run instead of all thousands of them" did not break the build. When someone noticed a week later, that was not fun to clean up.

Tests that test nothing useful are surprisingly easy to write, and lead to a similar result -- even in cases where you do run them! That's why it makes some sense to test the tests (or at least, check their bug-detecting quality).

Re: How to Write Unit Tests for Logging

#19

Post by gnusty_gnurc » Wed, Nov 11, 2020, 2:57 PM UTC

u801e wrote: post #8

It seems that the boundaries of what a unit test should cover have never been clearly defined. In my opinion, unit testing should only cover logic that does not have side effects (like writing to disk or communicating over a network). If you want to test external side effects, you should write functional or integration tests for them.

Unit tests are fine for testing side effects if you've written the code properly with proper interfaces, etc.

You need to test that you're using the contracts you've made, and integration tests verify that everything works as expected when both sides are complying with the contract.

I'm saying that if I've abstracted out the filesystem into an interface the distinction of asserting against whether a method was called on that interface and asserting a side-effect is somewhat trivial. I've made it so that side-effects must be handled in integration tests.

Re: How to Write Unit Tests for Logging

#20

Post by marcosdumay » Wed, Nov 11, 2020, 2:59 PM UTC

valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

I agree, but...

> Logging usually is so easy that it is an all- or nothing thing.

By all means, have the integration test that verifies that logs are produced at all (on each mechanism, if you support more than one). And it's a bonus if that test makes the deployment (or installation if that's your thing) fail.