Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database mocks are not worth it

shayon.dev

Post reply on HN
Print view View original

61–70 of 268 posts

Re: Database mocks are not worth it

#61

Post by fatbird »

senex wrote: post #4

Does anyone have experience making tests against real databases fast? I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations. I’d love to hear more from anyone that’s solved for some of these constraints!

Not sure if this is still a valid approach, but on a large Django site with a lot of unit tests, the cumulative setup/teardown/reset cycles was killing us. We found that setting each test to be wrapped with a transaction that was aborted on teardown, caused the per test cleanup to drop radically. We also kept a canned database for testing so that running the test suite didn't have a large startup penalty to populate the database with test fixtures. Keeping that ready db between runs also sped things up a lot for devs.

Re: Database mocks are not worth it

#62

Post by Volundr »

Lvl999Noob wrote: post #26

I faced problems with flaky unit tests that used a common unit test database for the whole project. Since all the tests ran at once, the tests would sometimes fail because of concurrency issues. I never got the time to look too deeply into it. I wonder if there's a database that works sorta like git, giving one "commit" point at the start of every test and then "branching off" so each test can do its own thing, make…

Datomic more or less works like this. I wasn't terribly impressed in general, but it did make tests a breeze.

Unless you use Clojure though it's probably not a real option.

Re: Database mocks are not worth it

#63

Post by pdimitar »

phendrenad2 wrote: post #33
pdimitar wrote: post #17

Earlier quoted context omitted.

I don't know what kind of magic fairy dust Django is but I've found the differences between SQLite and PostgreSQL too big to be worth it, in at least 3 other programming languages that are not Python. Sounded good at first but we were quickly overwhelmed with false positives and just opted for Postgres in a VM (this was before Docker was a thing).

This is a good thing to be aware of when choosing a database, but I think most of the time people just reach for Postgres because it's the "standard".

PostgreSQL stops many data bugs at the door due to being so strict -- something many programmers start rediscovering is a good thing by choosing stricter programming languages with time.

I love SQLite to bits but the test harness I have to put around my apps with it is a separate project in itself.

Re: Database mocks are not worth it

#65

Post by pdimitar »

withinboredom wrote: post #23
shayonj wrote: post #15

Earlier quoted context omitted.

+1 for this being common knowledge but there is still decent bit mocking that happens IME.

Mocks are great for testing specific code paths, but if you need to do that, there is usually a better way of doing it. Mocks hide contractual incongruities. For example, a function that returns either "fizz" or "buzz" that gets mocked out. If it gets changed to also return "bar", and you forget to update the mock, you've got a time-bomb in your code.

Completely agreed, that's why I have advocated for periodically (once a month) revisiting the test suites with them by doing re-captures of real data. There are frameworks that help with this.

Is it a chore? Absolutely. But peace of mind is important.

Re: Database mocks are not worth it

#66

Post by nwatson »

astine wrote: post #42

Testing against a real database is an example of integration testing. Using mocks is for unit testing. Ideally, you want to do both. Unit testing entails isolating particular components for testing which is important because it let's you be more precise in what exactly you're testing. Using mocks also makes it easier to run automated tests because it means you don't need to have a database or credentials handy during…

Then just use a database for unit testing as well.

Re: Database mocks are not worth it

#67

Post by 3pt14159 »

senex wrote: post #4

Does anyone have experience making tests against real databases fast? I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations. I’d love to hear more from anyone that’s solved for some of these constraints!

Tests are embarrassingly parallel. If you can split up the load for CI then at the limit the slowest response is your slowest running test. I wish more tooling would expose this out of the box, but with some effort it is possible.

Re: Database mocks are not worth it

#68

Post by fcmgr »

I've had some good experience with a mix of those approaches, maybe not using mocks per se, but an "in-memory database implementation" (just a wrapper around the hash map that implements the same behaviors as a repository that deals with a real database) on one hand, and testcontainers on the other. (Still, using an in-memory db is way better than mocking, the tests are not coupled to the implementation details/the underlying model).

For simple use cases where I mostly just read/write from the database and don't expect any of those issues mentioned in the article (constraint violations or concurrency issues, because the application is quite simple tbh, plus I already have some testcontainers based integration tests for the components that deal with a real db and I reuse them) writing tests with an in-memory db implementation is quite nice - the setup is simple and running the whole test suite is instantaneous (literally something like 1-2s for a couple thousand tests, I don't need any framework for those kind of tests).

And on the other hand if I'm relying on something like an optimistic/pesimisstic lock or a framework feature, I will write a test with the real thing, using test containers. Also have a pretty good coverage with the components that deal with queues and databases specifically with testcontainers. And on top of that just a few e2e flows written using testcontainers as well.

Re: Database mocks are not worth it

#69

Post by pdimitar »

nerdponx wrote: post #48
pdimitar wrote: post #5

I thought this was common knowledge and that it became even easier after Docker became a thing? Mocks are wishful thinking incarnate most of the time, though here and there they are absolutely needed (like 3rd party APIs without sandbox environments, or quite expensive API, or most of the time: both). Just pick a task runner -- I use just[0] -- and make a task that brings up both Docker and your containers, then run…

I've had a nice experience using Docker Compose for this, although you might still want a task runner wrapper around it. Podman Compose should work fine as well.

Exactly what I do. Having a local `docker-compose.yml` that helps bring up a dev and test environments is one of my first action items when hired. Paid huge dividends, many times.

Re: Database mocks are not worth it

#70

Post by brettgriffin »

I'd be really surprised if most applications that rely on databases can't just use a containerized image of their production database in any environment, including CI. In fact, the only example I can really think of is when the production database can't be Dockerized (e.g. a proprietary SaaS database like BigQuery or Snowflake).

I'm working on a project that has ~22,000 tests that operate on Docker container of the same version of Postgres used in production. In CI it completes all the database tests in under a minute.

Post reply on HN