Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multiple assertions are fine in a unit test

stackoverflow.blog

Post reply on HN
Print view View original

91–100 of 348 posts

Re: Multiple assertions are fine in a unit test

#91

Post by jmillikin »

masklinn wrote: post #88
jmillikin wrote: post #77

Earlier quoted context omitted.

GoogleTest (C++) and Go's built-in testing framework both support non-fatal assertions. They're used for code like this: assert_eq(list.len(), 1) expect_eq(list[0].username, "jdoe") expect_eq(list[0].uid, 1000) The idea being that if multiple properties are incorrect, then all of them will be printed out to the test log.

That seems like it'd easily get confusing, when the assertions are dependent. Which is often the case e.g. if the list is empty, testing the properties of the first item make no sense.

That's why the first check is a hard assertion (returning on error), and the others are soft (continuing on error).

If the list is empty, then the log will contain one error about the length being zero. If the list has one item but it has the wrong properties, the log will contain two errors.

Re: Multiple assertions are fine in a unit test

#92

Post by willjp »

I think part of the motivation for one assertion per test comes from the fact that you stop getting information from a test as soon as one assertion fails.

I think it was a guidance, like the SRP where you should be testing one thing in each test case. I also think a growing number of assertions might be a sign your unit under test is wearing too many responsibilities.

Maybe it’s better to say “few assertions, all related to testing one thing”

Re: Multiple assertions are fine in a unit test

#93

Post by superjan »

torginus wrote: post #89
TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

I remember writing a small .NET test library for that exact problem - You could pass in a lambda with a complex condition, and it evaluated every piece of the expression separately and pretty printed what part of the condition failed. So essentially you could write Assert(()=>width>0 && x + width And you would get: Assertion failed: x is 1500 width is 600 screenWidth is 1920 It used Expression to do the magic. Amazin…

Is this shared somewhere?

Re: Multiple assertions are fine in a unit test

#94

Post by FartyMcFarter »

magic_hamster wrote: post #51
TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having seen tests atrocious enough where this is not only possible, but the common case. Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Now imagine…

> You might know which line failed the test, but not always.

If that's the case, the test framework itself is severely flawed and needs fixing even more than the tests do.

There's no excuse to have an assert function that doesn't print out the location of the failure.

Re: Multiple assertions are fine in a unit test

#95

Post by masklinn »

jmillikin wrote: post #91
masklinn wrote: post #88

Earlier quoted context omitted.

That seems like it'd easily get confusing, when the assertions are dependent. Which is often the case e.g. if the list is empty, testing the properties of the first item make no sense.

That's why the first check is a hard assertion (returning on error), and the others are soft (continuing on error). If the list is empty, then the log will contain one error about the length being zero. If the list has one item but it has the wrong properties, the log will contain two errors.

> That's why the first check is a hard assertion (returning on error), and the others are soft (continuing on error).

See that's so completely unclear I utterly missed that there were two different calls there. Doesn't exactly help that the functions are the exact same length, and significantly overlap in naming.

Re: Multiple assertions are fine in a unit test

#96

Post by mjul »

ramraj07 wrote: post #58
magic_hamster wrote: post #51

Earlier quoted context omitted.

> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having seen tests atrocious enough where this is not only possible, but the common case. Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Now imagine…

So because some idiot somewhere wrote a 100 assertion unit test we should ban anyone from writing even 2 assertions in one test?

Just use you best judgement and you will be fine.

Patterns are context specific advice about a solution and its trade-offs, not hard rules for every situation.

Notice how pattern books will often state the relevant Context, the Problem itself, the Forces influencing it, and a Solution.

(This formulaic approach is also why pattern books tend to be a dry read).

Re: Multiple assertions are fine in a unit test

#97

Post by torginus »

TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

xUnit is terrible. It has a horrible culture, the author seems to have a god complex. It is overly complex and opinionated in the worst way possible.

Many times I searched for 'how do I do something with xUnit' and found a github issue with people struggling with the same thing, and the author flat out refusing to incorporate the feature as it was against his principles.

Other times I found that I needed to do was override some core xUnit class so it would do the thing I wanted it to do - sounds complex, all right lets see the docs. Oh there are none, 'just read the source' according to the author.

Another thing that bit us in the ass is they refused to support .NET Standard, a common subset of .NET Framework and Core, making migration hell.

Re: Multiple assertions are fine in a unit test

#98

Post by z9znz »

Testing is hard when the code it tests is OOP, has mutations everywhere, and has big functions that do too many things.

It's practically impossible to thoroughly test such code with one assertion per test; it would mean having dozens of tests just for one object method. Correspondingly, the fixtures/factories/setup for tests would balloon in number and complexity as well to be able to setup the exact circumstance being tested.

But the example in TFA is, imo, bad because it is testing two entirely different (and unrelated) layers at once. It is testing that a business logic delete of a thing works correctly, and that a communication level response is correct. Those could be two separate tests, separating the concerns, and resulting in simpler code to reason about and less maintenance effort in the future.

We want to know if the DeleteAsync(address) behaves correctly. Actually we want to know if DeleteReservation() works, irrespective of the async requirement. Testing whether AnythingAsync() works is something that is already done at the library or framework level, and we probably don't need to prove that it works again.

Write a test for DeletReservation() which tests if a valid reservation gets deleted. Write another related test to ensure that a non-existent or invalid reservation does not get deleted, but rather returns some appropriate error value. That's two, probably quite simple tests.

Now somewhere higher up, write the REST API tests. ApiDelete()... a few tests to establish that if an API delete is called, and the business logic function it calls internally returns a successful result, then does the ApiDelete() return an appropriate response? Likewise if the business logic fails, does the delete respond to the API caller correctly?

Re: Multiple assertions are fine in a unit test

#99

Post by Macha »

TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

I've noticed the opposite in a Java codebase I work in. Tests where the test is assertEquals(toJson(someObject), giantJsonBlobFromADifferentFile). Of course the test runner has no idea about formatting strings that happen to be json, so I end up having to copy these out into an editor, formatting them and eyeballing the difference, or for even larger ones having to save them out to files and diff them. And of course most of the fields in the mock aren't relevant to the class under test, so I'd trade them out for 5-6 targeted asserts for the relevant fields happily.

The problem is, since it's a legacy codebase, there's many fields which are only tested incidentally by this behaviour, by tests that actually aren't intending to test that functionality.

Re: Multiple assertions are fine in a unit test

#100

Post by johtso »

danuker wrote: post #52
olex wrote: post #42

Earlier quoted context omitted.

Hm. I think my main issue there is not the speed, but rather seeing the whole picture at once. You mentioned you use this pattern to test regular expressions; say you modify the regexp in question with some new feature requirement, and now the very first of a dozen test inputs fails. You fix it, but then each one of the following keeps failing, and you can only find an elegant solution that works for all of them afte…

In my experience, from doing some TDD Katas[0] and timing myself, I found coding slower and more difficult when focusing on multiple examples at once. I usually even comment out all the failing tests but the first one, after translating a bunch of specifications into tests, so I see the "green" when an example starts working. Maybe it would be easier to grok multiple regex examples than algorithmic ones, but at least…

Again pytest makes things so much nicer in this regard. Having to comment things out sucks.

With pytest you can use the -x flag to stop after the first test failure.

Even better you can use that in combination with -lf to only run the last failed test.

Post reply on HN