Database mocks are not worth it
shayon.dev
Database mocks are not worth it
1–10 of 268 posts
Re: Database mocks are not worth it
#2For tests, first use a real database. If you can't because the database just doesn't want to (ahem mongo ahem) then use a fake. If you can't use a fake, stop lying and use a fake.
Mocks for databases are extremely brittle and complicated.
Re: Database mocks are not worth it
#3I've found that replacing the database with in memory SQLite for tests is a sweet spot. Almost as fast as a mock, catches a lot of database issues. And it's really easy to do if you're using something like Django that makes automatically generating database migrations easy.
It won't help you with database specific differences. But there should be very few of those if you're using a framework that abstracts away the database. Like Django.
Re: Database mocks are not worth it
#4Does anyone have experience making tests against real databases fast?
I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations.
I’d love to hear more from anyone that’s solved for some of these constraints!
Re: Database mocks are not worth it
#5I thought this was common knowledge and that it became even easier after Docker became a thing?
Mocks are wishful thinking incarnate most of the time, though here and there they are absolutely needed (like 3rd party APIs without sandbox environments, or quite expensive API, or most of the time: both).
Just pick a task runner -- I use just[0] -- and make a task that brings up both Docker and your containers, then run your test task, done. Sure it's a bit fiddly the first time around but I've seen juniors conquer that in a day maximum and then your tests actually work with the real world 99% of the time.
Mocks in general are rarely worth it, the DB ones: 10x so.
Re: Database mocks are not worth it
#6For tests, first use a real database. If you can't because the database just doesn't want to (ahem mongo ahem) then use a fake. If you can't use a fake, stop lying and use a fake. Mocks for databases are extremely brittle and complicated.
Ahem MongoDB? I must admit, I've never understood what MongoDB's real use case is supposed to be. Whenever I've looked at it, there has always been a better alternative. When I've had to use it, I've had to debug performance problems that wouldn't have existed with alternative solutions.
It sounds cool. But running software isn't about sounding cool.
A decade ago, it was really clear. As https://aphyr.com/posts/284-call-me-maybe-mongodb explains, MongoDB didn't really work. But they've fixed that. So now it runs acceptably accurately. I just don't know why I'd ever want to.
Re: Database mocks are not worth it
#7Does anyone have experience making tests against real databases fast? I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations. I’d love to hear more from anyone that’s solved for some of these constraints!
A real database should not be slow. Even with our tests running against a hosted SQL Server on a separate server, the database is never the slow part. For other tests, we run with the same database in a local Docker container with Docker Compose and it is fast and isolated/resettable.
Most tests should be unit tests, which are super fast. Integration and UI tests that might use the database should be fewer and if the database is slow, it might be related to your specific application or unoptimized database queries, our database calls are usually < 10ms
Re: Database mocks are not worth it
#8Yeah, I ran in very much the same issues. Setting up a database for unit tests can be even more of a bother though (especially for CI pipelines, I don't want to have to run a full database there), so I took a middle road, and use my ORM to throw up a temporary in-memory SQLite database that is mostly similar to our actual database. Each unit tests scaffolds its own database, and deletes it when it's done.
That allows me to somewhat mock the database within unit tests, while still being able to test database-specific errors that might occur, like foreign key errors.
Re: Database mocks are not worth it
#9This risks unleashing a bit of a "religious" war.
I agree that tests should occur in the environment they are to be run in, when ever possible. Otherwise not...
There is no science in testing. There are no standardised definitions. We agree it is good, we often ignore the downsides (they exist - there are costs to tests)
I have struck examples where the testing frameworks of a language (Dart in this case) ruled out using a live HTTPS client to test. I kept getting 404 errors running it against my (test) site. I made a bug report. The good folks at Google (who I am sure are very smart - if a little stupid) had decided that the HTTPS client would always return 404 when run as part of the testing framework.
I argued, but apparently I am not so smart (I am less stupid, IMO). The HTTPS client testing had to either occur outside the test framework, or I rewrite the HTTPS client, or I wrote a mock up of some sort, all things I would have to pay for with time and effort - for no gain.
I get the point about mocking. But I am in business. I am not interested in always testing in isolation, (sometimes, sure), I want my tests to depend on the other parts of my business.
If the database goes wrong and my Gui tool test discovers it, that is a good thing, not a bad thing
Re: Database mocks are not worth it
#10This is one of many reasons why "isolating" I/O within your application is so important, whether that means pushing it to the edges or otherwise carefully corralling it in known-critical areas where it can't be removed.
A static type system that assists with that design style is nice as well. It's really useful to be able to look at a function and know right away what I/O it performs, if any. This is probably my #1 missing feature in Python, now that static type hints work well enough for most purposes.
That said, setting up and maintaining a database test harness in the developer environment can be really annoying and time consuming. Even moreso when your database is a proprietary cloud thing (e.g. Snowflake). But I have never ever regretted spending the time to set it up, whereas I have definitely regretted not having it.
Sometimes it's unavoidable that your "unit" tests need to access the database to be able to properly work out the "unit" under test. Watching those tests go from failing to passing is such a sweet feeling, it makes the pain of maintaining the test harness feel worthwhile. And then of course when you hit the actual integration tests in CI, everything almost always passes, because your unit tests are that good.