291–300 of 348 posts
Earlier quoted context omitted.
It's not nice to mock people "doing real http calls in unit tests," even if they deserve it.
For mocking, the parent comment means "swapping out the external calls with dummy calls", not "laughing at the developer"
> The excellent book xUnit Test Patterns describes a test smell named Assertion Roulette. It describes situations where it may be difficult to determine exactly which assertion caused a test failure. How is that even possible in the first place? The entire job of an assertion is to wave a flag saying "here! condition failed!". In programming languages and test frameworks I worked with, this typically includes providi…
I think the advantage of having an assertion per test is that it makes sure that all of your assertions are executed. In a lot of test frameworks (that use exceptions for assertions for example) the first assertion fail will stop the test. That doesn't mean you have to duplicate code, you can deal with it in other ways. In Junit I like to use @TestFctory [1] where I'll write most of the test in the Factory and then e…
The issue with multiple assertions for unit tests is that they hide errors by failing early extending the time to fix (fail at Assert 1, fix, re-run, fail at Assert 2, fix...). If you want multiple assertions you should have a single assertion that can report multiple issues by returning an array of error strings or something. The speed increase of multiple assertions vs multiple tests is usually tiny, but if you do have a long unit test then that's basically the only reason I can think of to use multiple assertions.
Earlier quoted context omitted.
I'd add one more: clearly document what determines the order in which tests are run. On the one hand, running tests in any order should produce the same result, and would in any decent test suite. On the other hand, if the order is random or nondeterministic, it's really annoying when 2% of PRs randomly fail CI, not because of any change in the code, but because CI happened to run unrelated tests in an unexpected ord…
> On the one hand, running tests in any order should produce the same result, and would in any decent test suite. Therefore the tool should run the tests in random order, to flush out the non-decent tests. IMHO.
Earlier quoted context omitted.
A unit tests tests one unit. And integration tests covers more than one unit. I think everyone agrees with that, but nobody has defined unit. The longer I program the more I am convinced that the larger your unit the better. The unit tests is a statement that you will never refactor across this line, and that eliminates a lot of flexibility that I want. It turns out that debugging failed integration tests is easy,the…
I recently went to the effort of trying to work out where the term unit test came from in some desperate effort to find what a unit was meant to be. After much googling and buying or ancient text books I hit a dead end. At this point I think "unit" is just noise that confuses people into making distinctions that don't exist.
Earlier quoted context omitted.
Yup, it's why I built `just-tap` [1] which trys to minimise as much magic that a lot of these frameworks try to "help" you with. 1. https://github.com/markwylde/just-tap
Here are a few mistakes I've seen in other frameworks: - Make it possible to disable timeouts. Otherwise, people will need a different runner for integration, long running (e.g., find slow leaks), and benchmark tests. At that point, your runner is automatically just tech debt. - It is probably possible to nest before and afters, and to have more than one nesting per process, either from multiple suites, or due to cla…
This way the framework can kill the test if it doesn't see one of these messages in a short amount of time.
This fixed our timeout issues. We had tests that took too long, specially in debug builds and we'd end up having to set too large a timeout. Now though, we can keep the timeout for the heartbeat really short and our timeout issues have mostly gone away.
Earlier quoted context omitted.
You should write an episode of Seinfeld. I was a TL on a project and I had two "eng" on the project that would make test with a single method and then 120 lines of Tasmanian Devil test cases. One of those people liked to write 600 line cron jobs to do critical business functions. This scarred me.
> One of those people liked to write 600 line cron jobs to do critical business functions. I was a long-time maintainer of Debian's cron, a fork of Vixie cron (all cron implementations I'm aware of are forks of Vixie cron, or its successor, ISC cron). There are a ton of reasons why I wouldn't do this, the primary one being is that cron really just executes jobs, period. It doesn't serialize them, it doesn't check for…
Earlier quoted context omitted.
Sometimes, it takes a lot less code to test a particular code by doing it in a long test with multiple assertions. Migrate up -> assert ok -> rollback 1 -> assert ok -> rollback 2 -> assert ok I don’t see much benefit to breaking it up, and you’re testing state changes between each transition, so the entire test is useful and simpler, shorter, and clearer than the alternative.
This can be a path where things do go bad. Let's say thus test pattern is a success and then is replicated for many tests. Now, the schema or migration changes. A small change there now breaks the entire test suite. At this point the number of failing tests only indicates how many hours you will be fixing assertions. Another failure mode is when test scaffolding builds up. Imagine that migrate up part becoming multip…
I wouldn’t actually write tests for migrations themselves.
These aren't unit tests, they are integration tests (API calls to an external system). Integration tests have had multiple assertions forever (look at how Cypress works where each test step is an assertion), the idea of single assertions are for unit tests. The issue with multiple assertions for unit tests is that they hide errors by failing early extending the time to fix (fail at Assert 1, fix, re-run, fail at Asse…
Even when I do have multiple failing tests, I focus on and fix them one test at a time.
Earlier quoted context omitted.
I think the advantage of having an assertion per test is that it makes sure that all of your assertions are executed. In a lot of test frameworks (that use exceptions for assertions for example) the first assertion fail will stop the test. That doesn't mean you have to duplicate code, you can deal with it in other ways. In Junit I like to use @TestFctory [1] where I'll write most of the test in the Factory and then e…
This is a feature. I want it to fail fast so I can see the first issue not crawl the stack for error logs.