Multiple assertions are fine in a unit test • Hacker News Board

Re: Multiple assertions are fine in a unit test

#331

Post by dmitriid » Sun, Nov 06, 2022, 7:06 PM UTC

RHSeeger wrote: post #327

dmitriid wrote: post #316

Earlier quoted context omitted.

> If I had to test all the use cases for each function, all combined together, there number of tests would grow by the multiplication of the partitions of each one Why would they? Do these edge cases not appear when the caller is invoked? Do you not test these edge cases and the behavior when the caller is invoked? As an example: you tested that your db layer doesn't fail when getting certain data and returns respons…

Unit one - returns a useful test for each type of error condition that can occur (N). Test that, for each type of error condition that can occur. One test for each error condition. Unit two - calls unit one - test that, if unit one returns an error, it is treated appropriately. One test, covers all error conditions because they're all returned the same way from Unit one. Unit three - same idea as unit one If you were…

> If you were to test the behavior of unit one _through_ units 2 and 3, you'd need 2*N tests.

There are only two possible responses to that:

1. No, there are not 2*N tests because unit 3 does not cover, or need, all of the behavior and cases that flow through those units. Then unit testing unneeded behaviors is unnecessary.

2. Unit 3 actually goes through all those 2*N cases. So, by not testing them at the unit 3 level you have no idea that the system behaves as needed. Literally this https://twitter.com/ThePracticalDev/status/68767208615275315...

> You're missing the point that you don't need to test "the exact same scenarios for the combination of units", because the partitions of

This makes no sense at all. Yes, you've tested those "inputs/outputs" in isolation. Now, what tests the flow of data? That unit 1 outputs data required by unit 2? That unit 3 outputs data that is correctly propagated by unit 2 back to unit 1?

Once you start testing the actual flow... all your unit tests are immediately entirely unnecessary because you need to test all the same cases, and edge cases to ensure that everything fits together correctly.

So, where I would write a single functional test (and/or, hopefully, an integration test) that shows me how my system actually behaves, you will have multiple tests for each unit, and on top of that you will still need a functional test, at least, for the same scenarios.

Re: Multiple assertions are fine in a unit test

#332

Post by RHSeeger » Sun, Nov 06, 2022, 8:13 PM UTC

dmitriid wrote: post #331

RHSeeger wrote: post #327

Earlier quoted context omitted.

Unit one - returns a useful test for each type of error condition that can occur (N). Test that, for each type of error condition that can occur. One test for each error condition. Unit two - calls unit one - test that, if unit one returns an error, it is treated appropriately. One test, covers all error conditions because they're all returned the same way from Unit one. Unit three - same idea as unit one If you were…

> If you were to test the behavior of unit one _through_ units 2 and 3, you'd need 2*N tests. There are only two possible responses to that: 1. No, there are not 2*N tests because unit 3 does not cover, or need, all of the behavior and cases that flow through those units. Then unit testing unneeded behaviors is unnecessary. 2. Unit 3 actually goes through all those 2*N cases. So, by not testing them at the unit 3 lev…

> Once you start testing the actual flow... all your unit tests are immediately entirely unnecessary because you need to test all the same cases, and edge cases to ensure that everything fits together correctly.

You don't, but it's clear that I am unable to explain why to you. I apologize for not being better able to express what I mean.

Re: Multiple assertions are fine in a unit test

#333

Post by magic_hamster » Sun, Nov 06, 2022, 8:42 PM UTC

ambrozk wrote: post #330

magic_hamster wrote: post #61

Earlier quoted context omitted.

Not at all. It makes sense in some tests. I addressed the part asking how it's even possible to not know what happened. As for multiple asserts, that is really meaningless. The test case should test one thing. If it requires several asserts that's okay. But having a very long test function with a lot of assertions, is strongly indicating that you're testing more than one thing, and when the test fails it will be hard…

> when the test fails it will be harder to know what actually happened. Yeah, and if you write one assertion at a time, it will be harder to write the tests. Decreasing #assertions/test decreases the speed of test debugging while increasing the time spent writing non-production code. It's a tradeoff. Declaring that the optimal number of assertions per test is 1 completely ignores the reality of this tradeoff.

That's true and it boils down to what's acceptable in your team (or just you). I worked in some places where coverage was the only metric and in places where every single function had to have all cases covered, and testing took longer than writing the code.

As for me, I tend to write reasonable tests and cover several cases that guard the intended behavior of each function (if someone decides the function should behave differently in the future, a test should fail). One emerging pattern is that sometimes during testing I realize I need to refactor something, which might have been lost on me if I skimmed on tests. It's both a sanity check and a guardrail for future readers.

Re: Multiple assertions are fine in a unit test

#334

Post by tobyhinloopen » Sun, Nov 06, 2022, 9:31 PM UTC

fiddlerwoaroof wrote: post #231

tobyhinloopen wrote: post #185

Earlier quoted context omitted.

Not really, no. I like to use Jest’s toMatchObject to combine multiple assertions in a single assertion. If the assertion fails, the full object on both sides is shown in logs. You can easily debug tests that way. The only way to make it even possible is to do some eval magic or to use a pre-processor like babel or a typescript compiler plugin. But if you find something, Lemme know.

Well, Function.toString() should print the code of a lambda in JavaScript. So I think you could do it without a pre-processor: use Babel as a library to parse the body of the function; run each sub-expression separately and display the results.

Can you actually do that, since you don’t have the context the function is running in?

You can inspect the source, but functions don’t run in isolation so you cannot extract the source and run it somewhere else.

Simple example:

    var x = 0;
    var y = function() { x++; };

You can’t reliably execute function y with just y’s function body.

Maybe if you replace the function with another by injecting code into a copy of the source, but I’m not sure if that’s even possible.

Re: Multiple assertions are fine in a unit test

#335

Post by fiddlerwoaroof » Sun, Nov 06, 2022, 9:40 PM UTC

tobyhinloopen wrote: post #334

fiddlerwoaroof wrote: post #231

Earlier quoted context omitted.

Well, Function.toString() should print the code of a lambda in JavaScript. So I think you could do it without a pre-processor: use Babel as a library to parse the body of the function; run each sub-expression separately and display the results.

Can you actually do that, since you don’t have the context the function is running in? You can inspect the source, but functions don’t run in isolation so you cannot extract the source and run it somewhere else. Simple example: var x = 0; var y = function() { x++; }; You can’t reliably execute function y with just y’s function body. Maybe if you replace the function with another by injecting code into a copy of the s…

You can just document a constraint that these function must use the purely functional subset of JavaScript, which is enough for the sorts of assertions I typically write. Alternatively, you could constrain it to the subset that has no unbound variables or side effects.

Re: Multiple assertions are fine in a unit test

#336

Post by int_19h » Sun, Nov 06, 2022, 10:20 PM UTC

mollerhoj wrote: post #233

krona wrote: post #70

Earlier quoted context omitted.

You tested a postcondition. What about preconditions and invariants, do you have separate tests for those assertions too, or just not bother?

Please correct me if I'm wrong, but would a precondtion not just be the postcondition of the setup? Invariants would either have to be publically available and thus easily testable with similar methods, or, one would have to use assertions in the implemention. I try to avoid the latter, as it mixes implemations and 'test/invariants'. Granted, there are situations (usually in code that implements something very 'algor…

assertions are basically comments with teeth; why would you avoid them in any code, algorithmic or no?

Re: Multiple assertions are fine in a unit test

#337

Post by int_19h » Sun, Nov 06, 2022, 10:25 PM UTC

OJFord wrote: post #306

ljm wrote: post #163

Earlier quoted context omitted.

I’ve seen people take a dogmatic approach to this in Ruby without really applying any critical thought, because one assertion per test means your test is ‘clean’. The part that is glossed over is that the test suite takes several hours to run on your machine, so you delegate it to a CI pipeline and then fork out for parallel execution (pun intended) and complex layers of caching so your suite takes 15 minutes rather…

> I’ve seen people take a dogmatic approach to this in Ruby without really applying any critical thought, because one assertion per test means your test is ‘clean’. I can't speak for Ruby, but what I would call 'clean' and happily dogmatise is that assertions should come at the end, after setup and exercise. I don't care how many there are, but they come last. I really hate tests that look like: setup() exercise(but_…

Here's a trivial rewrite that satisfies your dogmatic requirement without any meaningful difference:

    setup()
    
    exercise(but_not_for_the_last_time)

    was_blah = (state.currently == blah)

    state.do_other()
    something(state)

    assert was_blah && state.now == blegh

In fact, this last version is worse, because if do_other() can fail if state wasn't blah, then what you'll get is the exception from that failure interrupting the test before the assert would have been reported.

Re: Multiple assertions are fine in a unit test

#338

Post by int_19h » Sun, Nov 06, 2022, 10:27 PM UTC

mollerhoj wrote: post #247

ljm wrote: post #163

Earlier quoted context omitted.

I’ve seen people take a dogmatic approach to this in Ruby without really applying any critical thought, because one assertion per test means your test is ‘clean’. The part that is glossed over is that the test suite takes several hours to run on your machine, so you delegate it to a CI pipeline and then fork out for parallel execution (pun intended) and complex layers of caching so your suite takes 15 minutes rather…

Yes, you got us rubyists there. :-( Its the unfortunate result of trying to avoid premature optimization and strive for clarity instead. Something thats usually sound advice. Enginnering decisions have tradeoffs. When the testsuite becomes too slow, it might be time to reconsider those tradeoffs. Usually though, I find that to road to fast tests is to reduce/remove slow things (almost always some form of IO) not to c…

I'd argue that it's the unfortunate result of Ruby being at the center of the Agile and XP scene back when it first became prominent (the manifesto etc) - because that scene is also where the more cultish varieties of TDD originated.

Re: Multiple assertions are fine in a unit test

#339

Post by int_19h » Sun, Nov 06, 2022, 10:29 PM UTC

batiste wrote: post #110

lodovic wrote: post #4

Wait - people are doing real http calls in unit tests, over a network, and complaining about multiple asserts in the test code?

I have met people that thinks HTTP calls are great! - "But it tests more things!" Well ok, but those are integration tests, not unit tests... It is unacceptable that a unit tests can fail because of external system...

It's far better to have integration tests without unit tests than the other way around.

Re: Multiple assertions are fine in a unit test

#340

Post by kragen » Sun, Nov 06, 2022, 10:57 PM UTC

ckastner wrote: post #318

kragen wrote: post #312

Earlier quoted context omitted.

I thought Dillon cron was the default cron in Slackware? Hard to be a more major Linux distribution than Slackware, in terms of historical impact if not current popularity.

Could be. Slackware is a popular name, but I would call it "niche" rather than a major distribution. Just my personal view, obviously.

I just confirmed with a Slackware user today, it still does use Dillon cron. I had a vague memory from before I switched from Slackware to Debian late last millennium.