Multiple assertions are fine in a unit test • Hacker News Board

Re: Multiple assertions are fine in a unit test

#191

Post by hedora » Sat, Nov 05, 2022, 2:58 PM UTC

TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

> I've never seen a case where it would be hard to tell which assertion failed.

There are a set of unit testing frameworks that do everything they can to hide test output (junit), or vomit multiple screens of binary control code emoji soup to stdout (ginkgo), or just hide the actual stdout behind an authwall in a uuid named s3 object (code build).

Sadly, the people with the strongest opinions about using a "proper" unit test framework with lots of third party tooling integrations flock to such systems, then stack them.

I once saw a dozen-person team's productivity drop to zero for a quarter because junit broke backwards compatibility.

Instead of porting ~ 100,000 legacy (spaghetti) tests, I suggested forking + recompiling the old version for the new jdk. This was apparently heresey.

Re: Multiple assertions are fine in a unit test

#192

Post by 0x445442 » Sat, Nov 05, 2022, 2:59 PM UTC

Macha wrote: post #99

TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

I've noticed the opposite in a Java codebase I work in. Tests where the test is assertEquals(toJson(someObject), giantJsonBlobFromADifferentFile). Of course the test runner has no idea about formatting strings that happen to be json, so I end up having to copy these out into an editor, formatting them and eyeballing the difference, or for even larger ones having to save them out to files and diff them. And of course…

That test seems to be testing whether or not the library used to deserialize json works. I don’t think that’s valid unless the code base you are working on is Gson or Jackson or the like.

Assuming that’s not the case and you’re interested in the state of two object graphs then you just compare those, not the json string they deserialize to.

Re: Multiple assertions are fine in a unit test

#193

Post by magic_hamster » Sat, Nov 05, 2022, 2:59 PM UTC

FartyMcFarter wrote: post #94

magic_hamster wrote: post #51

Earlier quoted context omitted.

> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having seen tests atrocious enough where this is not only possible, but the common case. Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Now imagine…

> You might know which line failed the test, but not always. If that's the case, the test framework itself is severely flawed and needs fixing even more than the tests do. There's no excuse to have an assert function that doesn't print out the location of the failure.

Even if the framework is fine, you can see something like an elaborate if-else tree, or even a try-catch block, and after it's all done, there's a condition check with `fail()`. So the point of failure could be manually detached from the actual point of failure.

Granted, this is not the way to do things. But it happens anyway.

Re: Multiple assertions are fine in a unit test

#194

Post by mrits » Sat, Nov 05, 2022, 3:03 PM UTC

bluGill wrote: post #183

mrits wrote: post #174

Earlier quoted context omitted.

22 assertions in a test is a lot better than 22 separate tests that fail for the same reason.

They are the same. I don't care, as I'll fix them one at a time, and if the fix happens to fix more than one great.

Fixing one at a time like that is a good way to get into an endless cycle. Zoom out a bit and make a plan before you start coding.

Re: Multiple assertions are fine in a unit test

#195

Post by wccrawford » Sat, Nov 05, 2022, 3:04 PM UTC

iLoveOncall wrote: post #7

wccrawford wrote: post #2

First off, I do put more than 1 assertion in a test. But it definitely leads to situations where you have to investigate why a test failed, instead of it just being obvious. Like the article, I test 1 thing per test, but sometimes that means multiple assertions about the outcome of a test. IMO there's no point in checking that you got a response in 1 test, and then checking the content/result of that response in anot…

> IMO there's no point in checking that you got a response in 1 test, and then checking the content/result of that response in another test. The useful portion of that test is the response bit. If I understood this part correctly, you are making the dangerous assumption that your tests will run in a particular order.

No, I definitely am not making that assumption. With a bad response, but a good response code, 1 test would fail and the other would succeed, no matter the order. I just don't think that that valid response code is a useful test on its own. It's much better with both assertions in the same test, unless you have some reason to think that response code failure would signify something special on its own.

Re: Multiple assertions are fine in a unit test

#196

Post by xeromal » Sat, Nov 05, 2022, 3:06 PM UTC

beezlewax wrote: post #171

torginus wrote: post #89

Earlier quoted context omitted.

I remember writing a small .NET test library for that exact problem - You could pass in a lambda with a complex condition, and it evaluated every piece of the expression separately and pretty printed what part of the condition failed. So essentially you could write Assert(()=>width>0 && x + width And you would get: Assertion failed: x is 1500 width is 600 screenWidth is 1920 It used Expression to do the magic. Amazin…

Is there something like this available for javascript?

I'm not aware of any related to testing, but perhaps you could use something like this in tandem with some tests to pull it off?

https://github.com/gvergnaud/ts-pattern

Re: Multiple assertions are fine in a unit test

#197

Post by z9znz » Sat, Nov 05, 2022, 3:08 PM UTC

commandlinefan wrote: post #166

z9znz wrote: post #98

Testing is hard when the code it tests is OOP, has mutations everywhere, and has big functions that do too many things. It's practically impossible to thoroughly test such code with one assertion per test; it would mean having dozens of tests just for one object method. Correspondingly, the fixtures/factories/setup for tests would balloon in number and complexity as well to be able to setup the exact circumstance bei…

> when the code it tests is OOP In my experience when code isn’t OOP, that means all static functions with static (I.e. global) data which isn’t hard to test, it’s actually impossible because you can’t mock out the static data.

I didn't downvote you, but I have a hard time either understanding your meaning or imagining the scenario you describe. Can you give an example?

OOP functions are usually harder to test because they expect complete objects as arguments, and that tends to require a lot more mocking or fixtures/factories to setup for the test.

FP functions typically operate on less complex and more open data structures. You just construct the minimum thing necessary to satisfy the function, and the test is comparatively simple. None of this has anything to do with global data. Using global data from within any functions is generally a bad idea and has nothing to do with FP or OOP.

Re: Multiple assertions are fine in a unit test

#198

Post by tsss » Sat, Nov 05, 2022, 3:09 PM UTC

TeMPOraL wrote: post #133

Macha wrote: post #99

Earlier quoted context omitted.

I've noticed the opposite in a Java codebase I work in. Tests where the test is assertEquals(toJson(someObject), giantJsonBlobFromADifferentFile). Of course the test runner has no idea about formatting strings that happen to be json, so I end up having to copy these out into an editor, formatting them and eyeballing the difference, or for even larger ones having to save them out to files and diff them. And of course…

I had similar case recently, in C++. I ended up spending a few hours writing a simple JSON differ - a bit of code that would parse two strings into a DOM object graph using a rapidjson, and then walk down them simultaneously - basically, I implemented operator== which, instead of terminating early, recorded every mismatch. Then, I packaged it into a Google Test matcher, and from now on, the problem you describe is go…

> I think it's really important for programmers to learn to help themselves. If there's something that annoys you repeatedly, you owe it to yourself and others to fix it.

It's a cultural problem. _I_ can do that, but my colleagues will just continue to write minimum effort tests against huge json files or database dumps where you have no idea why something failed and why there are a bunch of assertions against undocumented magic numbers in the first place. It's like you're fighting against a hurricane with a leaf blower. A single person can only do so much. I end up looking bad in the daily standup because I take longer to work on my tickets but the code quality doesn't even improve in a measurable way.

Re: Multiple assertions are fine in a unit test

#199

Post by RHSeeger » Sat, Nov 05, 2022, 3:10 PM UTC

bluGill wrote: post #181

RHSeeger wrote: post #153

Earlier quoted context omitted.

> Testing lots of things at the same time means strictly speaking you're writing an integration test rather than a unit test There's nothing wrong with integration tests, but they're not unit tests. It's fine to have both, but the requirements for a good unit test and those for a good integration test diverge. The title of this post, at least, was specific to unit tests.

A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the…

> The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want.

I certainly don't see it as that. I see it as "this is the smallest thing I _can_ test usefully". Mind you, those do tend to correlate, but they're not the same thing.

Re: Multiple assertions are fine in a unit test

#200

Post by dxdm » Sat, Nov 05, 2022, 3:12 PM UTC

ronnier wrote: post #148

rhdunn wrote: post #129

I view it more as "only test one operation per unit test". If that needs multiple asserts (status code, response content, response mime type, etc.) to verify, that is fine. IIUC, the guideline is so that when a test fails you know what the issue is. Therefore, if you are testing more than one condition (missing parameter, invalid value, negative number, etc.) it is harder to tell which of those conditions is failing,…

I’ve come to the conclusion that none of this matters for most parts of a system. I worked in the most horrendous code and systems you can imagine but it turned into a multi billion dollar company. Then everyone starts talking about code quality and rewrites etc and new features stall as beautiful systems are written and high test coverage met and competing companies surpass us and take market share with new and bett…

I've worked on shitty code with shitty tests that ran the core of the business. Even while doing that, it was horrible to work with, held important features back and drove talented people away, leaving everything to stagnate in a "this works enough" state. When the wind changed, it was hard to turn the ship around, important people got nervous, and things got into a bad spiral.

None of this is the failure of code and tests alone; but both can be indicative of the structural health and resilience of the wider situation.