Multiple assertions are fine in a unit test • Hacker News Board

Re: Multiple assertions are fine in a unit test

#241

Post by ckastner » Sat, Nov 05, 2022, 5:14 PM UTC

sitkack wrote: post #227

hedora wrote: post #191

Earlier quoted context omitted.

> I've never seen a case where it would be hard to tell which assertion failed. There are a set of unit testing frameworks that do everything they can to hide test output (junit), or vomit multiple screens of binary control code emoji soup to stdout (ginkgo), or just hide the actual stdout behind an authwall in a uuid named s3 object (code build). Sadly, the people with the strongest opinions about using a "proper" u…

You should write an episode of Seinfeld. I was a TL on a project and I had two "eng" on the project that would make test with a single method and then 120 lines of Tasmanian Devil test cases. One of those people liked to write 600 line cron jobs to do critical business functions. This scarred me.

> One of those people liked to write 600 line cron jobs to do critical business functions.

I was a long-time maintainer of Debian's cron, a fork of Vixie cron (all cron implementations I'm aware of are forks of Vixie cron, or its successor, ISC cron).

There are a ton of reasons why I wouldn't do this, the primary one being is that cron really just executes jobs, period. It doesn't serialize them, it doesn't check for load, logging is really rudimentary, etc.

A few years ago somebody noticed that the cron daemon could be DoS'ed by a user submitting a huge crontab. I implemented a 1000-line limit to crontabs thinking "nobody would ever have 1000-line crontabs". I was wrong, quickly received bug reports.

I then increased it to 10K lines, but as far as I recall, users were hitting even that limit. Crazy.

Re: Multiple assertions are fine in a unit test

#242

Post by hbrn » Sat, Nov 05, 2022, 5:17 PM UTC

jeremyjh wrote: post #159

ronnier wrote: post #152

Earlier quoted context omitted.

It’s debt. When you can’t add new features quickly because you have nightmarish tests to fix and you spend more time on the tests than the product, I’d say it’s debt. Especially with the insane mocking setups.

Yes, those are the bad tests I was referring to. NOT having tests greatly increases the debt burden of your production code because you cannot refactor with any confidence and so you simply won’t.

> Yes, those are the bad tests I was referring to

This is the No True Scotsman issue with testing. When it fails, you just disregard the failure as "bad tests". But any company that has anything that resembles a testing culture will have a good amount of those "bad tests". And this amount is way higher than people are willing to admit.

> you cannot refactor with any confidence

Anecdotally, I've had way more cases where I wouldn't refactor because too many "bad tests" were breaking, not because I lacked confidence due to lack of tests.

There are many things beyond tests that allow you refactor with confidence: simple interfaces, clear dependency hierarchy, modular design, etc. They are way more important than tests.

Tests are often a last resort when all of the above is a disaster. When you're at a place where you need tests to keep your software stable you are probably already fucked, you're just not willing to recognize it.

You shouldn't have zero tests, but tests should be treated as debt. The fewer tests you need to keep your software stable, the better your architecture is. Huge number of tests in a codebase is typically a signal of shitty architecture that crumbles without those crutches.

Re: Multiple assertions are fine in a unit test

#243

Post by kazinator » Sat, Nov 05, 2022, 5:18 PM UTC

At work, I put in a change which allows multiple assertions in a C testing framework. There is no exception handling or anything.

A macro like

  EXPECT_ASSERT(whatever(NULL));

will succeed if whatever(NULL) asserts (e.g. that its argument isn't null). If whatever neglects to assert, then EXPECT_ASSERT will itself assert.

Under the hood it works with setjmp and longjmp. The assert handler is temporarily overridden to a function which performs a longjmp which changes some hidden local state to record that the assertion went off.

This will not work with APIs that leave things in a bad state, because there is no unwinding. However, the bulk of the assertion being tested are ones that validate inputs before changing any state.

It's quite convenient to cover half a dozen of these in one function, as a block of six one-liners.

Previously, assertions had to be written as individual tests, because the assert handler was overriden to go to a function which exits the process successfully. The old style tests are then written to set up this handler, and also indicate failure if the bottom of the function is reached.

Re: Multiple assertions are fine in a unit test

#244

Post by nesarkvechnep » Sat, Nov 05, 2022, 5:24 PM UTC

Are people so entrenched in dogmas that we need “Stop doing X” type articles every few months?

Re: Multiple assertions are fine in a unit test

#245

Post by kazinator » Sat, Nov 05, 2022, 5:24 PM UTC

TeMPOraL wrote: post #16

> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…

Think about the personality of someone who is so dissatisfied with the lack of verbosity in his test suites, that he needs a side project of writing a book about unit testing. Of course they will advocate testing one assertion per function, and make up nonsense to justify their recommendation.

Secretly, they would have the reader write 32 functions to separately test every bit of a uint32 calculation, only refraining from that advice due to the nagging suspicion that it might be loudly ridiculed.

Re: Multiple assertions are fine in a unit test

#246

Post by zzzeek » Sat, Nov 05, 2022, 5:25 PM UTC

"A foolish consistency is the hobgoblin of little minds"

- whoever

I definitely write tests with multiple assertions, the rule I try to follow is that the test is testing a single cause/effect. that is, a single set of inputs, run the inputs, then assert as many things as you want to ensure the end state is what's expected. there is no problem working this way.

Re: Multiple assertions are fine in a unit test

#247

Post by mollerhoj » Sat, Nov 05, 2022, 5:25 PM UTC

ljm wrote: post #163

alkonaut wrote: post #116

An assert message says what went wrong, and on which code line. How on earth does it help to make just one? The arrange part might take seconds for a nontrivial test and that would need to be duplicated both in code and execution time to make two asserts. If you painstakingly craft a scenario where you create a rectangle of a specific expected size why wouldn’t it be acceptable to assert both the width and height of…

I’ve seen people take a dogmatic approach to this in Ruby without really applying any critical thought, because one assertion per test means your test is ‘clean’. The part that is glossed over is that the test suite takes several hours to run on your machine, so you delegate it to a CI pipeline and then fork out for parallel execution (pun intended) and complex layers of caching so your suite takes 15 minutes rather…

Yes, you got us rubyists there. :-( Its the unfortunate result of trying to avoid premature optimization and strive for clarity instead. Something thats usually sound advice.

Enginnering decisions have tradeoffs. When the testsuite becomes too slow, it might be time to reconsider those tradeoffs.

Usually though, I find that to road to fast tests is to reduce/remove slow things (almost always some form of IO) not to combine 10 small tests into one big.

Re: Multiple assertions are fine in a unit test

#248

Post by seadan83 » Sat, Nov 05, 2022, 5:26 PM UTC

mannykannot wrote: post #104

dd82 wrote: post #54

Earlier quoted context omitted.

one assert per test seems... as you said, indicative of zealotry. if you already have your object there, why not test for the changes you expect? So you have one test that indicates that a log error is outut. then another that tests that the property X in the return from the error is what you expect. then another test to determine that propery Y in return is what you expect? that to me is wasteful, unclear, bloated.…

Furthermore, if you have a one-assertion rule, some bright spark will realize he can write a single assertion that checks for the conjunction of all the individual postconditions. That's one way to get dogma-driven assertion roulette, as you will not know which particular error occurred.

If all assertions are at the end of the test, then yes. Sometimes this can be made nice with custom matchers, eg:

assertThat(fooReturningOptional(), isPresentAndIs(4))

Or

assertThat(shape, hasAreaEqualTo(10))

Or

AssertThat(polygonList, hasNoIntersections())

Custom matchers can go off the deep end really easily. One of those cases of learn the principle, then learn when it does not apply

Re: Multiple assertions are fine in a unit test

#249

Post by superjan » Sat, Nov 05, 2022, 5:27 PM UTC

TrianguloY wrote: post #177

superjan wrote: post #122

Earlier quoted context omitted.

I think the issue is that you’ll always have one of those teammates who see this as an excuse to test the entire happy flow and all its effects in a single test case. I think what you want is reasonable, but how do you agree when it is no longer reasonable?

If you logic depends on that happy path, make a test for it. But as I explained in another comment that test should not justify the lack of individual feature tests, which should not only test the happy path but other corner cases too. On my company we developers usually create white-box unitary/feature tests (we know how it was implemented, so we check components knowing that). But then we have an independent QA tea…

Sounds like a fine approach, and I wasn’t criticizing. Mostly I was pondering out loud why people come up with blanket statements what good tests should look like.

Re: Multiple assertions are fine in a unit test

#250

Post by [deleted] » Sat, Nov 05, 2022, 5:31 PM UTC

zzzeek wrote: post #246

"A foolish consistency is the hobgoblin of little minds" - whoever I definitely write tests with multiple assertions, the rule I try to follow is that the test is testing a single cause/effect. that is, a single set of inputs, run the inputs, then assert as many things as you want to ensure the end state is what's expected. there is no problem working this way.

[deleted]