41–50 of 82 posts
Earlier quoted context omitted.
Writing unit tests is debatable?
Unit tests are code, and all code has a cost. Writing unit tests without any benefit is harmful behaviour. Real world example: testing dumb getters/setters.
So, disclaimer: I'm about to focus on the finger instead of where it's pointing. Because that's what I feel like jabbering about right now. TFA isn't just about databases; it makes some interesting points on test practice in general, and is well worth a read. Anyway, A story I've seen all too often when mocking the database: A large development effort goes into creating test infrastructure. And then there end up bein…
You write a comprehensive test of integration tests against the real object.
You then set it up so those same tests can run both the in-memory and real version.
Any differences should show up for the interactions you have specified, and tests should fail.
You can write sociable unit tests against the in-memory version, knowing it matches the same behaviour as closely as you have specified in the tests.
What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…
Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less…
I disagree. A failing unit test doesn't even necessarily indicate that an error was introduced: if a unit doesn't do what it's supposed to but the user doesn't see that, then an error wasn't introduced. Sure, when a unit test fails there's often an error, but if an end-to-end test fails, there's always an error--E2E tests are testing from the user's perspective, so what they're testing is actually errors. (This is assuming that both the unit tests and the E2E tests are correctly written).
You're positioning unit tests as a debugging tool, but I'd argue that there are much better debugging tools: REPLs and debuggers give you a lot more information than a unit test, and allow you to ask new questions quickly.
I don't want to come across as being anti-unit tests. On the contrary, I think unit tests are highly valuable. But I don't think the value comes from gathering information, debugging, or even catching bugs (in most cases). I think the value comes from a few things:
1. TDD forces you to design units for reuse from the start. Immediately you're using the code in two contexts: the application and the unit test. So right away your code is inherently reusable (in a binary sense) because de-facto you've re-used it. Reusability is more complicated than that (it's really more of a spectrum than a binary) but having at least two uses from the beginning pushes you toward the reusable side of the spectrum.
2. Unit tests act as living documentation for units. It is often unclear by reading code what the code does, because production concerns such as performance and security can lead you to do things in seemingly complicated ways. But unit tests don't have these concerns (at least not in the same way) so you can write code in unit tests that clearly communicates what a unit does. And unit tests don't fall out of sync with code like plain text documentation does.
3. TDD is incredibly motivating. Moving red->green on a quick cycle takes advantage of the dopamine reward system to increase productivity.
Our production database is postgres.
We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we could keep running the tests. We knew that we could run them against a postgres db, but the development time to rewrite our CI test runner to spin up a fresh database was not worth it.
Recently we migrated from github + Jenkins to gitlab + our own gitlab-runners in AWS. During that switch, we prioritized testing against a Postgres DB and got it all running in a containerized way that spins up at fresh DB for every test run. The tests are slower but the runners scale horizontally so we don't mind queueing up as many merge requests as we need to and deployments through Gitlab environments are a big improvement over our Jenkins deployment job, so we still ship as often as we want.
Now our biggest testing issue is keeping fixtures up to date.
This also makes it infinitely easier to coordinate complex schema changes. Each developer can sort it all out on their local branch before we even know about it. If we had to share some common test database, this would become a much more painful process.
Also, I do not believe in using mock layers for the database interactions. Our service implementations are tightly-coupled with their backing datastore. This is the only way we are able to make SQLite a viable storage medium for a high-throughput business application. As a consequence, testing our services in absence of their concrete datastores would be an extremely disingenuous endeavor for us.
All you have is a shared place, mocking is essentially putting up a fence around it to prevent testing into that boundary
If your database is immutable and is indexed for time travel, you can rewind or fast-forward your database into your desired state, if your database supports speculative writes you can even build up a non-committable state
I'm sure Rich Hickey will have a talk on this in relation to databases
So, disclaimer: I'm about to focus on the finger instead of where it's pointing. Because that's what I feel like jabbering about right now. TFA isn't just about databases; it makes some interesting points on test practice in general, and is well worth a read. Anyway, A story I've seen all too often when mocking the database: A large development effort goes into creating test infrastructure. And then there end up bein…
> scads of bugs that weren't caught by the unit tests Which is fine! Unit tests will never catch all the bugs - neither will type safety. Neither will code reviews. Neither will manual testing. But unit tests do catch the kinds of bugs that unit tests are good at catching, which eases the burden on the manual testers.
Commenter implies that bugs occurred in code that was assumed to be tested and correct (according to the test specs) because it did have tests. Which is decidedly Not Fine.
Now, would I consider them to be “unit tests” in this case? Probably not. But the label you decide to slap on the test doesn’t change the fact that a spec was written and code was tested against it and passed (falsely) due to mocking the db.
What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…
Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less…
Unit tests on code with dependencies (whether injected so that they can be mocked, or directly referenced so that they end up more like mini integration tests) are less excellent. They're brittle, inhibit refactoring, and either don't test as much as you think they do (if mocking dependencies) or are slow (if not mocking).
The further up the testing pyramid you go, the less work it is to refactor things, because you don't need to rewrite as many tests. OTOH test are more complex to write and take longer to run.
And now I get to my point: I don't think the blanket statement of "your largest set of test should be ... unit testst that run quickly" is well-founded. There are trade-offs, and they shouldn't be trivially waved aside.
You should never mock anything. Test against the same dependencies that exist in production. If that's impossible (i.e. you get charged for the backends or you are controlling physical objects), then generalize the program to support alternative backends and frequently test only those that can work in the test environment, using some more ad-hoc methodologies for the others. Also, in general, if you need to change yo…
I don't see how this relates to mocking at all. You can write good or bad tests this way regardless of mocking.
For a long time, we ran our Django test suite against an in-memory SQLite DB. It was super fast, which encouraged more tests to be written and a CI/CD process that allowed everyone to confidently ship code often. Our production database is postgres. We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we cou…