Multiple assertions are fine in a unit test • Hacker News Board

Re: Multiple assertions are fine in a unit test

#261

Post by lifeisstillgood » Sat, Nov 05, 2022, 6:17 PM UTC

ckp95 wrote: post #158

lifeisstillgood wrote: post #120

Earlier quoted context omitted.

Yes parameterised testing. Just all the packages i have tried seemed to make it awkward to impossible - hence metaclasses

Does this do what you want? https://github.com/ckp95/pytest-parametrize-cases @parametrize_cases( Case("handle null", age=None, x="foobar"), Case("don't allow under 13s", age=11, x="foobar"), Case("or old age pension", age=77, x="wobble"), ... # as many as you want ) def test_lots_of_ways_to_fail(age, x): with pytest.raises(ValueError): function_under_test(age, x)

Just wanted to pop in and say ckp95 actually mailed me this reply in case I missed it. The extra effort kinda restored my faith in humanity - giving a shit about strangers problems matters these days. Nice one.

cf https://youtu.be/wUyI3ssmj_U

Re: Multiple assertions are fine in a unit test

#262

Post by dvh » Sat, Nov 05, 2022, 6:28 PM UTC

My problem with assertions is that they can be turned off. I've seen things that could never happen happen in production on daily basis.

Re: Multiple assertions are fine in a unit test

#263

Post by alkonaut » Sat, Nov 05, 2022, 6:31 PM UTC

flavius29663 wrote: post #190

alkonaut wrote: post #116

An assert message says what went wrong, and on which code line. How on earth does it help to make just one? The arrange part might take seconds for a nontrivial test and that would need to be duplicated both in code and execution time to make two asserts. If you painstakingly craft a scenario where you create a rectangle of a specific expected size why wouldn’t it be acceptable to assert both the width and height of…

That can be considered one logical assertion though. You're asserting the size of the rectangle. You can even extract an assert helper function AssertRectangleSize(20,10)

Exactly. But if I assert N properties of an object is that then one assert logically for any N? At what point does it stop?

Applying any rule dogmatically is often bad, and this is no exception. The is that we don’t like lacking rules. Especially it goes to hell when people start adding code analysis to enforce it, and then developers start writing poor code that passes the analysis.

One assert imo isn’t even a good starting point that might need occasional exceptions.

Re: Multiple assertions are fine in a unit test

#264

Post by zmj » Sat, Nov 05, 2022, 7:07 PM UTC

jasonhansel wrote: post #257

hedora wrote: post #214

Earlier quoted context omitted.

Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to cla…

I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected ord…

I’ll disagree with this. Every time I’ve seen that, the interference between tests was also possible between requests in production. I’d rather my test framework give me a 2% chance of noticing the bug than 0%.

Re: Multiple assertions are fine in a unit test

#265

Post by teddyh » Sat, Nov 05, 2022, 7:12 PM UTC

sdenton4 wrote: post #232

bluGill wrote: post #181

Earlier quoted context omitted.

A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the…

Michael Pollan was right: Write tests, not too many, mostly integration.

Contrary viewpoint: Integrated Tests Are A Scam (J.B. Rainsberger):

https://www.youtube.com/watch?v=fhFa4tkFUFw

Re: Multiple assertions are fine in a unit test

#266

Post by thomaslangston » Sat, Nov 05, 2022, 7:15 PM UTC

jasonhansel wrote: post #257

hedora wrote: post #214

Earlier quoted context omitted.

Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to cla…

I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected ord…

Test order should be random, so that the ability to run them in parallel and distribute them across multiple hosts is not lost by missing enforcement of test isolation.

Re: Multiple assertions are fine in a unit test

#267

Post by Jabbles » Sat, Nov 05, 2022, 7:20 PM UTC

mrits wrote: post #174

magic_hamster wrote: post #51

Earlier quoted context omitted.

> How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". I envy you for never having seen tests atrocious enough where this is not only possible, but the common case. Depending on language, framework and obviously usage, assertions might not be as informative as providing the basic functionality of failing the test - and that's it. Now imagine…

22 assertions in a test is a lot better than 22 separate tests that fail for the same reason.

With 22 separate tests you have the possibility of knowing that only a subset of them fail. Knowing which fail and which pass may help you debug.

In Go, in general, tests fail and continue, rather than causing the test to stop early, so you can tell which of those 22 checks failed. Other languages may have the option to do something similar.

https://pkg.go.dev/testing#T.Fail

Re: Multiple assertions are fine in a unit test

#268

Post by schubart » Sat, Nov 05, 2022, 7:21 PM UTC

jasonhansel wrote: post #257

hedora wrote: post #214

Earlier quoted context omitted.

Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to cla…

I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected ord…

> On the one hand, running tests in any order should produce the same result, and would in any decent test suite.

Therefore the tool should run the tests in random order, to flush out the non-decent tests. IMHO.

Re: Multiple assertions are fine in a unit test

#269

Post by teddyh » Sat, Nov 05, 2022, 7:24 PM UTC

Kwpolska wrote: post #14

littlecranky67 wrote: post #9

Earlier quoted context omitted.

> Is it weird that not only have I never heard of the "rule" this post argues against This "rule" is known mostly because it is featured in the "Clean Code" book by Robert C. Martin (Uncle Bob). You should have heard of it ;)

The book is so full of bad advice I'm not surprised this "rule" comes from there as well.

As MikeDelta reports¹, the book doesn’t actually say that.

I’ve come to learn to completely disregard any non-specific criticism of that book (and its author). There is apparently a large group of people who hate everything he does and also, seemingly, him personally. Everywhere he (or any of his books) is mentioned, the haters come out, with their vague “it’s all bad” and the old standard “I don’t know where to begin”. Serious criticism can be found (if you look for it), and the author himself welcomes it, but the enormous hate parade is scary to see.

1. https://news.ycombinator.com/item?id=33480517

Re: Multiple assertions are fine in a unit test

#270

Post by david_allison » Sat, Nov 05, 2022, 7:38 PM UTC

Macha wrote: post #99

TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

I've noticed the opposite in a Java codebase I work in. Tests where the test is assertEquals(toJson(someObject), giantJsonBlobFromADifferentFile). Of course the test runner has no idea about formatting strings that happen to be json, so I end up having to copy these out into an editor, formatting them and eyeballing the difference, or for even larger ones having to save them out to files and diff them. And of course…

This can be improved, it'd be worth Googling for a better solution than what you have.

https://github.com/skyscreamer/JSONassert seems decent.

but it can be done from scratch in a few hours (I'd recommend this if you have 'standardized' fields which you may want to ignore):

Move to a matcher library for assertions (Hamcrest is decent), and abstract `toJSON` into the a matcher, rather on the input.

This would change the assertion from:

`assertEquals(toJson(someObject), giantJsonBlobFromADifferentFile)`

to:

`assertThat(someObject, jsonEqual(giantJsonBlobFromADifferentFile))`

The difference here is subtle: it allows `jsonEqual` to control the formatting of the test failure output, so on a failure you can:

* convert both of the strings back to JSON

* perform a diff, and provide the diff in the test output.

Decent blog post on the topic: https://veskoiliev.com/use-custom-hamcrest-matchers-to-level...