Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Write Unit Tests for Logging

principal-it.eu

Post reply on HN
Print view View original

21–30 of 38 posts

Re: How to Write Unit Tests for Logging

#21

Post by parhamn »

geewee wrote: post #14
valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

Completely agree with this. I've wanted to test type A logs, and often heard people saying that "you should never test logs" No - you should never test most types of logs - sometimes logs, particularly in error situations, are part of the output of a function.

This sounds like a nomenclature thing. I’m sure any reasonable developer would agree it makes sense to test expected output of a program that isn’t strictly for debugging purposes (type A mentioned above), it’s just that is usually called output not “logs”.

Re: How to Write Unit Tests for Logging

#22

Post by brlewis »

u801e wrote: post #8

It seems that the boundaries of what a unit test should cover have never been clearly defined. In my opinion, unit testing should only cover logic that does not have side effects (like writing to disk or communicating over a network). If you want to test external side effects, you should write functional or integration tests for them.

I don't think clearly defined boundaries are what's needed. People just need to keep in mind that the purpose of tests is to catch bugs. When I modify this code, what tests will tell me that it still does what it's supposed to? Keeping that question at the fore will work better than any cookbook.

Re: How to Write Unit Tests for Logging

#23

Post by ivanbakel »

valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

> Logs which break the program semantics when missing.

What's an example of this kind of logging? I can imagine perhaps legally-mandated audit logs for financial software, but that would be stretching the definition of "semantics" in my mind.

Re: How to Write Unit Tests for Logging

#24

Post by rixrax »

valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

Back in the day when everything was written in C/C++ it wasn’t unusual that seemingly well tested software had careless memcpy or sprintf left in the error/log execution path that then exposed that system to remote code execution.

Just wanted to throw this in as to discourage completely ignoring testing logging. Of course e.g. good static analysis should somewhat alleviate chance of surprises here.

Re: How to Write Unit Tests for Logging

#25

Post by sorokod »

valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

I would suggest a third type:

c) Logs that when present, "break" (using the term loosely) the application. The example here is when application is logging sensitive data, e.g. PII

Re: How to Write Unit Tests for Logging

#26

Post by closeparen »

pydry wrote: post #15
u801e wrote: post #8

It seems that the boundaries of what a unit test should cover have never been clearly defined. In my opinion, unit testing should only cover logic that does not have side effects (like writing to disk or communicating over a network). If you want to test external side effects, you should write functional or integration tests for them.

I wish more people believed this. The number of lines of code I've seen written of pointless unit tests is staggering. If your code is mostly passing data around and managing side effects a unit test is more like a parasite. It will couple to your code and fail noisily when refactoring but not provide any actual useful feedback.

My armchair psychoanalysis is that it’s precisely poor effort:reward ratio that makes people feel superior for writing and demanding them. Look at me, I am a responsible engineer who tests thoroughly, you childish cowboys are trying to wriggle out of doing your chores.

No, I’m still going to sweep the floor, just not with a toothpick.

Re: How to Write Unit Tests for Logging

#27

Post by xboxnolifes »

ivanbakel wrote: post #23
valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

> Logs which break the program semantics when missing. What's an example of this kind of logging? I can imagine perhaps legally-mandated audit logs for financial software, but that would be stretching the definition of "semantics" in my mind.

Maybe I'm misunderstanding the parent, but I interpreted as the same kind of "logging" as the standard output of a CLI application. You expect a grep tool to output result to standard output for the user to use. Or you would expect user-facing errors to occur if an sql client fails to connect to a server.

Re: How to Write Unit Tests for Logging

#28

Post by zoomablemind »

geewee wrote: post #14
valenterry wrote: post #5

There are two types of logging: a) Logs which break the program semantics when missing. That means, the program does not work "as expected" anymore for any directly or indirectly impacted enduser. b) Logs that have no semantic impact when missing. No one will notice that they are missing, unless there is any other problem. These logs are solely meant to better understand the program operations. Logs of type A should…

Completely agree with this. I've wanted to test type A logs, and often heard people saying that "you should never test logs" No - you should never test most types of logs - sometimes logs, particularly in error situations, are part of the output of a function.

> ...and often heard people saying that "you should never test logs"

It's surprising that it gets labeled as "logs". Don't we test the expected behavior of the _code_ under test?

Be it a business or an auxilliary code like logging, it's still code, some function/class/facility...well, any purpose added code.

Thus, I would test such logging facility for correctness; otherwise, the next time its output is needed, I want to be sure that it can be trusted.

Of course, using ready-made logging frameworks may help shift such burden of testing onto the framework's developer.

Re: How to Write Unit Tests for Logging

#29

Post by u801e »

gnusty_gnurc wrote: post #19
u801e wrote: post #8

It seems that the boundaries of what a unit test should cover have never been clearly defined. In my opinion, unit testing should only cover logic that does not have side effects (like writing to disk or communicating over a network). If you want to test external side effects, you should write functional or integration tests for them.

Unit tests are fine for testing side effects if you've written the code properly with proper interfaces, etc. You need to test that you're using the contracts you've made, and integration tests verify that everything works as expected when both sides are complying with the contract. I'm saying that if I've abstracted out the filesystem into an interface the distinction of asserting against whether a method was called…

> I'm saying that if I've abstracted out the filesystem into an interface the distinction of asserting against whether a method was called on that interface and asserting a side-effect is somewhat trivial.

But now you're enforcing a degree of coupling between your filesystem interface and your code. If you refactor the filesytem interface such that the overall logic hasn't changed, but the parameter order for one of the methods is updated, then you essentially double the amount of work you have to do (update the code and update the tests). If you just test the logic and not the interface, then you only have to update the tests that pertain to the logic itself.

You should read up on the concept of functional core/imperative shell.

Re: How to Write Unit Tests for Logging

#30

Post by u801e »

brlewis wrote: post #22
u801e wrote: post #8

It seems that the boundaries of what a unit test should cover have never been clearly defined. In my opinion, unit testing should only cover logic that does not have side effects (like writing to disk or communicating over a network). If you want to test external side effects, you should write functional or integration tests for them.

I don't think clearly defined boundaries are what's needed. People just need to keep in mind that the purpose of tests is to catch bugs. When I modify this code, what tests will tell me that it still does what it's supposed to? Keeping that question at the fore will work better than any cookbook.

> When I modify this code, what tests will tell me that it still does what it's supposed to?

While that's true, you don't want to end up in a situation where the test is enforcing coupling which makes it harder to refactor your code. So, for instance, if you have tests that assert whether the method tested calls another method with certain parameters in a certain order, then you're enforcing coupling between the two methods. If you refactor the code to have the first method return a result and then use that result to call the second method, then you also have to update the tests. But they shouldn't have to be updated since the overall logic hasn't changed. If, instead, you test the logic of each method individually, then you can refactor and not have to update the tests.

Post reply on HN