I mean, obviously. Obviously I'm not going to assert every header in their own unit test for a single request path in most cases. Assert what makes sense as a logical whole / unit.
Multiple assertions are fine in a unit test
41–50 of 348 posts
Re: Multiple assertions are fine in a unit test
#42Earlier quoted context omitted.
This seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate…
> your example stops on the first one that fails, and does not evaluate the others after that. I don't think this is a big problem; trying to focus on multiple examples at once is difficult. It might be a problem if tests are slow and you are forced to work on all of them at once. But in that case I'd try to make the tests faster (getting rid of network requests, disk/DB access by faking them away or hoisting to the…
Hm. I think my main issue there is not the speed, but rather seeing the whole picture at once. You mentioned you use this pattern to test regular expressions; say you modify the regexp in question with some new feature requirement, and now the very first of a dozen test inputs fails. You fix it, but then each one of the following keeps failing, and you can only find an elegant solution that works for all of them after seeing all the failures, having ran the test and modified the code a dozen times. Wouldn't it be nicer to see all fails right away and be able to find a solution to all of them, instead of fixing the inputs one-by-one?
Re: Multiple assertions are fine in a unit test
#43This is such a bad example because the level of testing is somewhat between unit and acceptance/functional testing. I can't tell at a glance if "api" is something that will hit a database or not.
The first part where he says you can check in the passing test of code that does nothing makes me twitch, but mainly because I see functional testing as the goal for a bit of functionality and unit tests as a way of verifying the parts of achieving that goal. Unit tests should verify the code without requiring integration and functional tests should confirm that the units integrate properly. I wouldn't recommend checking in a test that claims to verify that a deleted item no longer exists when it doesn't actually verify that.
Deciding on the granularity of actual unit tests is probably something that is best decided through trial and error. I think when you break down the "rules", like one assertion per test, you need to understand the goals. In unit testing an API I might have lots of tiny tests that confirm things like input validation, status codes, debugging information, permissions, etc. I don't want a test that's supposed to check the input validation code to fall because it's also checking the logged-in state of the user and their permission to reach that point in the code.
In unit tests, maybe you want to test the validation of the item key. You can have a "testItemKey" test that checks that the validation confirms that the key is not null, not an empty string, not longer than expected, valid base64, etc. Or you could break those into individual tests in a test case. It's all about the balance of ergonomics and the informativeness and robustness of the test suite.
In functional testing, however, you can certainly pepper the tests with lots of assertions along the way to confirm that the test is progressing and you know at what point it broke. In that case, the user being unable to log in would mean that testing deleting an item would not be worthwhile.
Re: Multiple assertions are fine in a unit test
#44Earlier quoted context omitted.
I've seen several test frameworks that don't abort a test case after the first failed assertion. When you get many tests each emitting multiple failures because one basic thing broke, the output gets hard to sort through. It's easier when the failures are all eager.
Which ones? I’ve used at least a dozen at this point, across C++, C#, JavaScript, Rust — and all of them throw (the equivalent of) exceptions on assertion failures.
My experience with testing framework was that they do all tests and then mark the ones that failed.
Re: Multiple assertions are fine in a unit test
#45I like the one assertion per unit test case rule. It makes me design cleaner code and write readable test cases. I also use object equality vs asserting on individual fields.
Re: Multiple assertions are fine in a unit test
#46Earlier quoted context omitted.
Which ones? I’ve used at least a dozen at this point, across C++, C#, JavaScript, Rust — and all of them throw (the equivalent of) exceptions on assertion failures.
My experience with testing framework was that they do all tests and then mark the ones that failed.
That has nothing to do with not knowing which assertion in a given test has failed.
Re: Multiple assertions are fine in a unit test
#47> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…
In phpunit you can send dataset through a test function . If you don't label them you will have a jolly time finding out which one of the sets caused the failure.
Re: Multiple assertions are fine in a unit test
#48Earlier quoted context omitted.
I use the following pattern for testing regexes: expected_positive = [ 'abc', 'def', ...] for text in expected_positive: self.assertTrue(matcher(text), f"Failed: {text}") Before I added the assertion error message, `f"Failed: {text}"`, it was quite difficult to tell WHICH example failed.
This seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate…
not really. one thing this is useful for is extracting out various attributes in an object when you really don't want to compare the entire thing. Or comparing dict attributes, and figuring which one is the incorrect one.
for example,
expected_results = {...}
actual_obj = some_intance.method_call(...)
for key, val in expected_results.items():
assert getattr(actual_obj, key) == val, f"Mismatch for {key} attribute"
Re: Multiple assertions are fine in a unit test
#49Earlier quoted context omitted.
I use the following pattern for testing regexes: expected_positive = [ 'abc', 'def', ...] for text in expected_positive: self.assertTrue(matcher(text), f"Failed: {text}") Before I added the assertion error message, `f"Failed: {text}"`, it was quite difficult to tell WHICH example failed.
That looks like a case where I would use a parameterized test rather than a for loop inside the test.
this has downsides if you're comparing attributes with a method result and checking whether said attrs match what you expect. Either you run each test N times for N attr comparisons, accepting the cost of setup/teardown, or do a loop and fire off an assert error with text on which comparison failed.
Since you already have the object right there, why not do the latter approach?
Re: Multiple assertions are fine in a unit test
#50Earlier quoted context omitted.
This seems like a test design issue to me. Best practice is to avoid for-each loops with assertions within tests - using parametrized tests and feeding the looped values as input is almost always a better option. Figuring out which one failed and why is one advantage it gives you in comparison. Another one is that all inputs will always be tested - your example stops on the first one that fails, and does not evaluate…
not really. one thing this is useful for is extracting out various attributes in an object when you really don't want to compare the entire thing. Or comparing dict attributes, and figuring which one is the incorrect one. for example, expected_results = {...} actual_obj = some_intance.method_call(...) for key, val in expected_results.items(): assert getattr(actual_obj, key) == val, f"Mismatch for {key} attribute" You…
Ok, in this case a parametrized test is not the best approach, I agree. But I would still want to avoid the for-each and "failing fast". One approach would be to gather the required attributes in an array or a struct of some sort, and then do a single assert comparison with an expected value, showing all the differences at once. However, this requires the assertion framework to be able to make such a comparison and return a nicely readable error message, ideally with a diff.