Live data from Hacker News

Multiple assertions are fine in a unit test

stackoverflow.blog

221–230 of 348 posts

Re: Multiple assertions are fine in a unit test

#221
I think two important requirements for good unit tests are that 1) If you look at a particular test, you can easily tell exactly what it's testing and what the expected behavior is, and 2) You are able to easily look at the tests in aggregate and determine whether they're covering all the behavior that you want to test.

To that end, I think a better guideline than "only have one assertion per test" would be "only test one behavior per test". So if you're writing a test for appending an element to a vector, it's probably fine to assert that the size increased by one AND assert that the last element in the vector is now the element that you inserted.

The thing I see people do that's more problematic is to basically pile up assertions in a single test, so that the inputs and outputs for the behavior become unclear, and you have to keep track of the intermediate state of the object being tested in your head (assuming they're testing an object). For instance, they might use the same vector, which starts out empty, test that it's empty; then add an element, then test that its size is one; then remove the element, test that the size is 0 again; then resize it, etc. I think that's the kind of testing that the "one assertion per test" rule was designed to target.

With a vector it's easy enough to track what's going on, but it's much harder to see what the discrete behaviors being tested are. With a more complex object, tracking the internal state as the tests go along can be way more difficult. It's a lot better IMO to have a bunch of different tests with clear names for what they're testing that properly set up the state in a way that's explicit. It's then easier to satisfy the above two requirements I listed.

I want to be able to look at a test and know exactly what I don't mind if a little bit of code is repeated - you can make functions if you need to help with the test set up and tear down.

Re: Multiple assertions are fine in a unit test

#222
I work to "assert one thing", and the assertions are added to help with debugging. I sometimes even assert starting conditions if they can at all change.

Without a "assert one thing", what tends to happen is there will be cut-and-paste between tests, and the tests overlap in what they assert. This means that completely unrelated tests will have an assertion failure when the code goes wrong.

When you do a refactor, or change some behavior, you have to change _all_ of the tests. Not just the one or two that have the thing you're changing as their focus.

Think of tests that over-assert like screenshot or other diff-based tests, they are brittle.

Re: Multiple assertions are fine in a unit test

#223
post #220

Earlier quoted context omitted.

TAP is better than some things, but it has some serious issues that I wrote about on my blog a while back - https://blog.urth.org/2017/01/21/tap-is-great-except-when-it... Basically it's nearly impossible to fully parse it correctly.

Is test2 a flag for TAP? If you have to pick one or the other, then you're breaking the common flow (human debugging code before pushing) so that management can have better reports. The right solution would be to add a environment variable or CLI parameter that told tap to produce machine readable output, preferably with a separate tool that could convert the machine readable junk to whatever TAP currently writes to…

Test2 is a Perl distribution that replaces a bunch of older test stuff. See https://metacpan.org/pod/Test2

But unlike TAP, it's fairly Perl-specific as opposed to just being an output format. I imagine you could adapt the ideas in it to Node but it'd be more complex than simply implement TAP in JS.

And yes, I think the idea of having different output formats makes sense. With Test2, the test _harness_ produces TAP from the underlying machine-readable format, rather than having the test code itself directly product TAP. The harness is a separate program that executes the tests.

Re: Multiple assertions are fine in a unit test

#224

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

I remember writing a small .NET test library for that exact problem - You could pass in a lambda with a complex condition, and it evaluated every piece of the expression separately and pretty printed what part of the condition failed. So essentially you could write Assert(()=>width>0 && x + width And you would get: Assertion failed: x is 1500 width is 600 screenWidth is 1920 It used Expression to do the magic. Amazin…

Related to this, for anyone not fully up to date on recent C# features there is also the CallerArgumentExpression [1], [2] feature introduced in C# 10. While it is not a pretty printer for an expression, it does allow the full expression passed from the call site as an argument value to be captured and used within the method. This can be useful for custom assert extensions.

For example:

    public void CheckIsTrue(bool value, [CallerArgumentExpression("value")] string? expression = null)
    {
        if (!value) 
        { 
            Debug.WriteLine($"Failed: '{expression}'"); 
        }
    }
So if you call like this: CheckIsTrue(foo != bar && baz == true), when the value is false it prints "Failed: 'foo != bar && baz == true'".

[1] https://learn.microsoft.com/en-us/dotnet/csharp/language-ref... [2] https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...

Re: Multiple assertions are fine in a unit test

#225

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

I remember writing a small .NET test library for that exact problem - You could pass in a lambda with a complex condition, and it evaluated every piece of the expression separately and pretty printed what part of the condition failed. So essentially you could write Assert(()=>width>0 && x + width And you would get: Assertion failed: x is 1500 width is 600 screenWidth is 1920 It used Expression to do the magic. Amazin…

I love using Unquote[0] in F# for similar reasons; it uses F#'s code quotations. Assuming the variables have been defined with the values you state, the assertion is written as:

  test  0 && x + width 
And part of the output is:

  width > 0 && x + width  0 && x + width 
[0]: https://github.com/SwensenSoftware/unquote

Re: Multiple assertions are fine in a unit test

#226

Earlier quoted context omitted.

It's not nice to mock people "doing real http calls in unit tests," even if they deserve it.

For mocking, the parent comment means "swapping out the external calls with dummy calls", not "laughing at the developer"

You failed a humor unit test.

Re: Multiple assertions are fine in a unit test

#227
post #191

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

> I've never seen a case where it would be hard to tell which assertion failed. There are a set of unit testing frameworks that do everything they can to hide test output (junit), or vomit multiple screens of binary control code emoji soup to stdout (ginkgo), or just hide the actual stdout behind an authwall in a uuid named s3 object (code build). Sadly, the people with the strongest opinions about using a "proper" u…

You should write an episode of Seinfeld.

I was a TL on a project and I had two "eng" on the project that would make test with a single method and then 120 lines of Tasmanian Devil test cases. One of those people liked to write 600 line cron jobs to do critical business functions.

This scarred me.

Re: Multiple assertions are fine in a unit test

#228
post #220

Earlier quoted context omitted.

Is test2 a flag for TAP? If you have to pick one or the other, then you're breaking the common flow (human debugging code before pushing) so that management can have better reports. The right solution would be to add a environment variable or CLI parameter that told tap to produce machine readable output, preferably with a separate tool that could convert the machine readable junk to whatever TAP currently writes to…

Test2 is a Perl distribution that replaces a bunch of older test stuff. See https://metacpan.org/pod/Test2 But unlike TAP, it's fairly Perl-specific as opposed to just being an output format. I imagine you could adapt the ideas in it to Node but it'd be more complex than simply implement TAP in JS. And yes, I think the idea of having different output formats makes sense. With Test2, the test _harness_ produces TAP fr…

What is this madness?

Nothing should have to be parsed. Write test results to sqlite, done. You can generate reports directly off those test databases using anything of your choice.

    your-program test-re.sqlite output.html

Re: Multiple assertions are fine in a unit test

#229
post #8

I haven't heard of the single-assertion thing in at least 10 years, probably 15. In the early 2000s, when I was starting out and doing .NET, it used to be something you'd hear in the community as a very general guideline, more like "there's something to be said about very focused tests, and too many assertions might be a smell." At the time, I got the impression that the practice had come over from Java and converted…

Like you I was surprised to hear this is a thing or is even controversial. Admittedly I've only been programming for about 10 years, but I haven't heard (or seen) this come up even one time. Every test I've ever seen has usually had multiple mutations and assertions, all of them testing the same premise.

Re: Multiple assertions are fine in a unit test

#230
I think it depends on the test. If you are checking that e.g. context.currentUser() returns current user name, email, id and roles I would probably wrote just 2 tests. One for the user attributes and one for checking roles.

jUnit provides a helpful assertAll(...) method that allows us to check multiple assertions without stopping and the first failed one.

I my tests I often use "thick" asserts like assertEqualsWithDiff(dtoA, dtoB), that compare 2 objects as a whole and prints property names with values that do not match. Not everyone likes this approach, but for me it is the best balance between time spend on the test and the benefit that I get from it.

Post reply on HN