Live data from Hacker News

Database mocks are not worth it

shayon.dev

221–230 of 268 posts

Re: Database mocks are not worth it

#221

Earlier quoted context omitted.

A hobby project of mine (in Elixir) uses SQLite as primary database. Each test runs in its own fully isolated SQLite database. No mocking (or transaction rolling back) needed. Most of these tests take less than 1ms to run (and when they take longer, it's because of something else). This kind of setup makes the usual Ecto Sandbox approach feel slow, but I do agree that the way Elixir approaches this is great!

Do you have a link you can share that demonstrates the details of this approach?

The entire project is open source, so sure! :D

I actually have two projects that use this approach, FeebDB (which is the library I wrote to manage a "one SQLite database per client" approach) and HackerExperience (a game under development that uses FeebDB).

The overall idea is simple:

1. Before tests start running, create a prop of each database.

2. The prop contains the "starting database" for each test. It may contain seed data (optional).

3. For each test, copy the prop and assign it a unique shard identifier (say, cp /props/user.db /test_data/user/874125.db).

4. The test knows the `shard_id` and can do whatever it wants with it; no one else will bother it.

5. Once ExUnit is finished, delete all shards.

Both projects follow a similar approach (I wrote it first in FeebDB and copied into HackerExperience, which has some sections commented out -- I need to clean up this part of the codebase).

For both projects, you will find steps 1/5 in `test/support/db.ex`, step 2 in `test/support/db/prop.ex` and steps 3/4 in `test/support/case/db.ex`.

- FeebDB: https://github.com/renatomassaro/FeebDB/

- HackerExperience: https://github.com/HackerExperience/HackerExperience/

Email is in profile in case you have follow up questions/comments :)

Re: Database mocks are not worth it

#222

Earlier quoted context omitted.

I haven't seen the "stored procs only"-style of DB development since the early 2000's, and I'd never go back to that.

Lambda functions would like to have a word...

At least with Lambda you can use mainstream programming languages. You can also have single lambdas that handle multiple endpoints or events. It’s a bit different.

Re: Database mocks are not worth it

#223
post #3

I've found that replacing the database with in memory SQLite for tests is a sweet spot. Almost as fast as a mock, catches a lot of database issues. And it's really easy to do if you're using something like Django that makes automatically generating database migrations easy. It won't help you with database specific differences. But there should be very few of those if you're using a framework that abstracts away the d…

As many others already stated, there are more than just small subtle differences that will bite you with this approach. Eventually forcing you to develop unnecessary compatibility abstractions or downgrade to less efficient lowest common denominator feature set to make it work. It's not worth the effort.

Especially since launching postgres is equally easy and fast as sqlite. Docker can help with sandboxing. What is left to gain? 100ms shorter startup time or keeping your unit test executables as single binaries? Irrelevant.

Re: Database mocks are not worth it

#224
post #207

Earlier quoted context omitted.

> How do you roll back a dozen stored procedures quickly? Liquibase rollback feature. The SQL changesets and rollback strategy is all defined in my .sql liquibase files and goes inside my git repo alongside my scala/python code the associated CICD automated integration tests + deployments. Blue Green deployment can handled via bidirectional active-active database replication across your two disaster recovery database…

I agree with Liquibase for source control of schema. But you don’t see how much harder this is for developers with feature flags in stored procedures, etc over standard git? There is tooling around feature flags for code I think we are in violent agreement though, for OLAP and analytics, I wouldn’t care as much. Because of the way that Redshift works (columnar store) and how it compiles and optimizes queries across a…

Thanks! Good point about integrating with feature flag tooling... I'll have to think more about that.

Re: Database mocks are not worth it

#225
post #219
post #216

Earlier quoted context omitted.

They used it in a startup where I used to work. Their data was fully relational, and they were doing the one thing that really really kills performance in mongodb: growing documents. Also it had no constraints so the data was all fucked up by the various bugs that were in the code over the years. Ah yes they used sharding. Probably there wouldn't have been a need for it if they had just used postgres, since the data…

PG and Mongo will vertically scale about the same depending on your queries. They were probably using sharding with tiny instances which is dumb. Also, large documents doesn't really hurt performance with mongo, except maybe with writes, or with large array fields due to replication implication.

Not large documents, but documents where the most common operation is to add a number to a list of numbers.

Re: Database mocks are not worth it

#226
post #213
post #146

Earlier quoted context omitted.

How do you write the expected return from an external API in your test setup?

You use a "VCR" module to record the real request and response and then play it back in your unit test. Boom you get the power of an integration test with the speed and determinism of a unit test. Then you turn off the VCR in CI to catch if/when the upstream API changes on you and now it's a real integration test. The problem with write tests, not too many, mostly integration is that unit tests are too damn good at g…

My question was a rhetorical one, I do use request-replayers and other techniques (I've been through a bunch in the past 20+ years as they were developed).

I was challenging the absolutism of "never use mocks", it's just another technique and can be applied easily in integration tests if the guidelines are well set in the team on how to not use them.

Also, I do not do external calls in the CI/CD pipeline, it inevitably makes tests brittle and flaky.

Re: Database mocks are not worth it

#227
post #223
post #3

I've found that replacing the database with in memory SQLite for tests is a sweet spot. Almost as fast as a mock, catches a lot of database issues. And it's really easy to do if you're using something like Django that makes automatically generating database migrations easy. It won't help you with database specific differences. But there should be very few of those if you're using a framework that abstracts away the d…

As many others already stated, there are more than just small subtle differences that will bite you with this approach. Eventually forcing you to develop unnecessary compatibility abstractions or downgrade to less efficient lowest common denominator feature set to make it work. It's not worth the effort. Especially since launching postgres is equally easy and fast as sqlite. Docker can help with sandboxing. What is l…

> Especially since launching postgres is equally easy and fast as sqlite

It's definitely not as fast to start postgres as it is to start sqlite. Pretty much inherently - postgres has to fork a bunch of processes, establishes network connectivity etc. And running trivial queries will always be faster with sqlite, because executing queries via postgres will require intra-process context switches.

That's not to say postgres is bad (I've worked on it for most of my career), but there just are inherent advantages and disadvantages of in-process databases vs out-of-process databases. And lower startup time and lower "dispatch" overhead are advantages of in-process databases.

Re: Database mocks are not worth it

#228

Earlier quoted context omitted.

> However you must test it via integration tests. that is not what you said. you wrote: "Don’t code with mocks period" > I differentiated between unit tests and integration tests. you did not. your original comment doesnt mention integration tests at all. i quote: "Structure your code such that it has a functional core and imperative shell. All logic is unit testable (functional core). All IO and mutation Is not unit…

https://chatgpt.com/share/6773b196-7da8-8001-accd-aefbfe4bac... Rarely used unless necessary is what I meant. The point of the integration test is to avoid mocking and I don’t want to argue a pedantic point with you and I also don’t like your attitude so I’m ending this section of the thread don’t bother replying.

hey man the least you could say us "yeah i wrote something that I didn't mean, my bad" when i called you out. instead you dug in, moved goalposts, and claimed you wrote shit you didn't.

it started with literally bad professional advice. junior developers reading these forums need to not have professional habits influenced by the sort of behavior on display by you here.

and a ChatGPT transcript? really? just do a simple search, you will find tons of articles advocating for mocking external apis in integration tests.

Re: Database mocks are not worth it

#229

Earlier quoted context omitted.

The production DB is MSSQL, and we scaffold it in SQLite through EF Core. The resulting SQLite DB is close enough to our production DB that we are able to catch invalid defaults, missing foreign keys, etc. in unit tests instead of later on in our testing pipeline, which helps massively in accelerating our development. It could be even better if SQLite would actually tell you which foreign key constraint failed instea…

Well, if you found a productive workflow then who am I to judge, right? However, if I was hired into your team tomorrow you'll have to fiercely fight with me over this: > And sure, we could probably refactor it to use transactions to shave a few seconds of running our test suite, but it'd add some additional mental complexity to our codebase to do so. Various languages and frameworks demonstrate that abstracting this…

Could you give an example or two of languages/frameworks that have demonstrated abstracting the transaction blocks away? I'm not sure I'm following so I think this will help.

Re: Database mocks are not worth it

#230
post #229

Earlier quoted context omitted.

Well, if you found a productive workflow then who am I to judge, right? However, if I was hired into your team tomorrow you'll have to fiercely fight with me over this: > And sure, we could probably refactor it to use transactions to shave a few seconds of running our test suite, but it'd add some additional mental complexity to our codebase to do so. Various languages and frameworks demonstrate that abstracting this…

Could you give an example or two of languages/frameworks that have demonstrated abstracting the transaction blocks away? I'm not sure I'm following so I think this will help.

I lost the link to one Golang library that I liked very much but here's the link to Elixir's Ecto: https://hexdocs.pm/ecto_sql/Ecto.Adapters.SQL.Sandbox.html (not the perfect one, sorry, but can't be bothered to look for a better resource). The TL;DR is as above: it uses the same DB but does parallel connections to it and in each you have a transaction that's ultimately rolled back.
Post reply on HN