Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Multiple assertions are fine in a unit test

stackoverflow.blog

Post reply on HN
Print view View original

341–348 of 348 posts

Re: Multiple assertions are fine in a unit test

#341

Post by ttkciar »

ckastner wrote: post #318
kragen wrote: post #312

Earlier quoted context omitted.

I thought Dillon cron was the default cron in Slackware? Hard to be a more major Linux distribution than Slackware, in terms of historical impact if not current popularity.

Could be. Slackware is a popular name, but I would call it "niche" rather than a major distribution. Just my personal view, obviously.

It always gives me a spell of cognitive dissonance when someone points out that Slackware is no longer a "major" distribution.

It used to be the major distribution. Funny how times change.

Re: Multiple assertions are fine in a unit test

#342

Post by hbrn »

jeremyjh wrote: post #272
hbrn wrote: post #234

Earlier quoted context omitted.

They are still debt since they don't directly contribute to product value: you can delete all your tests and your software will keep functioning. It doesn't mean it's a debt worth taking, though IME most companies are either taking way too much or way to little. Not treating tests as debt typically leads to over-testing, and it is way worse than under-testing. Also, what you're talking about (business-oriented requir…

If you can simply delete it that’s not debt, it’s just cost. My bank doesn’t let me delete my loan obligation and still live in my house.

You can simply burn the bank, no?

I've never seen a team where bad tests were simply deleted. They were always "fixed" instead.

Re: Multiple assertions are fine in a unit test

#343

Post by fomine3 »

beezlewax wrote: post #171
torginus wrote: post #89

Earlier quoted context omitted.

I remember writing a small .NET test library for that exact problem - You could pass in a lambda with a complex condition, and it evaluated every piece of the expression separately and pretty printed what part of the condition failed. So essentially you could write Assert(()=>width>0 && x + width And you would get: Assertion failed: x is 1500 width is 600 screenWidth is 1920 It used Expression to do the magic. Amazin…

Is there something like this available for javascript?

power-assert

Re: Multiple assertions are fine in a unit test

#344

Post by dmitriid »

RHSeeger wrote: post #332
dmitriid wrote: post #331

Earlier quoted context omitted.

> If you were to test the behavior of unit one _through_ units 2 and 3, you'd need 2*N tests. There are only two possible responses to that: 1. No, there are not 2*N tests because unit 3 does not cover, or need, all of the behavior and cases that flow through those units. Then unit testing unneeded behaviors is unnecessary. 2. Unit 3 actually goes through all those 2*N cases. So, by not testing them at the unit 3 lev…

> Once you start testing the actual flow... all your unit tests are immediately entirely unnecessary because you need to test all the same cases, and edge cases to ensure that everything fits together correctly. You don't, but it's clear that I am unable to explain why to you. I apologize for not being better able to express what I mean.

> You don't

If you don't, then you you have no idea if your units fit together properly :)

I've been bitten by this when developing microservices. And as I said in an edit above, it becomes less clear what to test in more monolithic apps and in GUIs, but in general the idea still holds.

Imagine a typical simple microservice. It will have many units working together:

- the controller that accepts an HTTP request

- the service layer that orchestrates data retrieved from various sources

- the wrappers for various external services that let you get data with a single method call

- a db wrapper that also lets you get necessary data with one method call

So you write extensive unit tests for your DB wrapper. You think of and test every single edge case you can think of: invalid calls, incomplete data etc.

Then you write extensive unit tests for your service layer. You think of and test every single edge case you can think of: invalid calls, external services returning invalid data etc.

Then you write extensive unit tests for your controller. Repeat above.

So now you have three layers of extensive tests, and that's just unit tests.

You'll find that most (if not all) of those are unnecessary for one simple reason: you never tested how they actually behave. That is, when the microservice is actually invoked with an actual HTTP request.

And this is where it turns out that:

- those edge cases you so thoroughly tested for the DB layer? Unnecessary because invalid and incomplete data is actually handled at the controller layer, or service layer

- or that errors raised or returned by service wrappers, or the db layer either don't get propagated up, or are handled by a generic catch all so that the call returns a nonsensical stuff like `HTTP 200: {error: "Server error"}`

- or that those edge cases actually exist, but since you tested them in isolation, and you didn't test the whole flow, the service just fails with a HTTP 500 error on invalid invocation

Or, instead, you can just write a single suite of functional tests that test all of that for the actual controllerservicewrappers flow covering the exact same scenarios.

Re: Multiple assertions are fine in a unit test

#345

Post by oskrim »

__alexs wrote: post #204
bluGill wrote: post #181

Earlier quoted context omitted.

A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the…

I recently went to the effort of trying to work out where the term unit test came from in some desperate effort to find what a unit was meant to be. After much googling and buying or ancient text books I hit a dead end. At this point I think "unit" is just noise that confuses people into making distinctions that don't exist.

See "Software Engineering at Google" https://abseil.io/resources/swe-book/html/ch11.html, the definition of a "small test".

This is roughly my definition of unit test: "tests run in a single process"

Re: Multiple assertions are fine in a unit test

#346

Post by LawTalkingGuy »

jasonhansel wrote: post #257
hedora wrote: post #214

Earlier quoted context omitted.

Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to cla…

I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected ord…

The tests are fatally broken. It means you can't trust them to properly check new work even.

The solution is to use random ordering and print the ordering seed with each run so it can be repeated when it triggers an error. Immediately halt all new work until randomly run tests don't have problems.

This isn't as bad as it sounds, generally it's a few classes of things that cause the interference which each will fix many tests. It's unlikely that the code actually has a 2%+ density of global-variable use, for example.

Re: Multiple assertions are fine in a unit test

#347

Post by johtso »

masklinn wrote: post #109
johtso wrote: post #100

Earlier quoted context omitted.

Again pytest makes things so much nicer in this regard. Having to comment things out sucks. With pytest you can use the -x flag to stop after the first test failure. Even better you can use that in combination with -lf to only run the last failed test.

> With pytest you can use the -x flag to stop after the first test failure. > Even better you can use that in combination with -lf to only run the last failed test. Fwiw `--sw` is much better for that specific use-case. `--lf` is more useful to run the entire test suite, then re-run just the failed tests (of the entire suite). IIRC it can have some odd interactions with `-x` or `--maxfail`, because the strange things…

Oh nice, didn't know about that flag!

Another option is to use a custom mark on the test you want to run, and then do something like "pytest -v -m onlyrunthis"

Re: Multiple assertions are fine in a unit test

#348

Post by OJFord »

int_19h wrote: post #337
OJFord wrote: post #306

Earlier quoted context omitted.

> I’ve seen people take a dogmatic approach to this in Ruby without really applying any critical thought, because one assertion per test means your test is ‘clean’. I can't speak for Ruby, but what I would call 'clean' and happily dogmatise is that assertions should come at the end, after setup and exercise. I don't care how many there are, but they come last. I really hate tests that look like: setup() exercise(but_…

Here's a trivial rewrite that satisfies your dogmatic requirement without any meaningful difference: setup() exercise(but_not_for_the_last_time) was_blah = (state.currently == blah) state.do_other() something(state) assert was_blah && state.now == blegh In fact, this last version is worse, because if do_other() can fail if state wasn't blah, then what you'll get is the exception from that failure interrupting the tes…

Exactly because that's 'without any meaningful difference' is why I don't like that either. I'm not obsessing over purely the 'assert' keyword as you perhaps think I am, it's the structure I don't like.
Post reply on HN