Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database mocks are not worth it

shayon.dev

Post reply on HN
Print view View original

91–100 of 268 posts

Re: Database mocks are not worth it

#91

Post by mrkeen »

fcmgr wrote: post #83
mrkeen wrote: post #74

Earlier quoted context omitted.

My take is that your business logic shouldn't know about your storage tech. (Like, dependency inversion 101 right?) > Still, using an in-memory db is way better than mocking, the tests are not coupled to the implementation details/the underlying model. Isn't this backwards? The fact that you've backed your storage with a HashMap (which is 100% what I shoot for too) means your service-under-test cannot know if it's ta…

I think we're talking about the same thing :D? The service/business logic/whatever you might want to call it interacts with the storage via an interface - in prodcution that interface is implemented by a component that can talk to a real SQL database, in tests I can just create a wrapper around a hash map and use that. EDIT: What I meant when I wrote that tests are not coupled to the underlying model/details is that…

> with a mock you have to explicitly specify "when called with this return that".

Riiiight, no I've always hated that Mockito way of doing things.

I find wrapping a hashmap avoids the need for explicit when-this-then-that bindings, because a hashmap already does the expected behaviour natively.

You can even 'integrate' your tests as deeply as you like, all standing on top of your poor hashmap. I.e. Http tests with unit test speed.

var controller = new Controller(new Service(new Repo(new HashMap()))) controller.POST(...); assert(controller.GET(...));

Re: Database mocks are not worth it

#92

Post by donjoe »

fcmgr wrote: post #28
pdimitar wrote: post #5

I thought this was common knowledge and that it became even easier after Docker became a thing? Mocks are wishful thinking incarnate most of the time, though here and there they are absolutely needed (like 3rd party APIs without sandbox environments, or quite expensive API, or most of the time: both). Just pick a task runner -- I use just[0] -- and make a task that brings up both Docker and your containers, then run…

I've had good experience with testcontainers ( https://testcontainers.com/ ) to do that sort of thing.

testcontainers is great. I struggled a bit with testcontainers due to the nature of one container per test which just felt too slow for writing gray/blackbox tests. The startup time for postgres was > 10 seconds. After a bit of experimenting, I am now quite happy with my configuration which allows me to have a snappy, almost instant testing experience.

My current setup:

- generate a new psql testcontainer _or_ reuse an existing one by using a fixed name for the container - connect to the psql container with no database selected - create a new database using a random database name - connect to the randomly generated database - initialize the project's tables - run a test - drop the database - keep the testcontainer up and running and reuse with next test

With this setup, most tests run sub-second;

Re: Database mocks are not worth it

#93

Post by maxbond »

zeroonetwothree wrote: post #58
rikthevik wrote: post #16

Earlier quoted context omitted.

> But there should be very few of those if you're using a framework that abstracts away the database. But I really want that database-specific behaviour. :) PostgreSQL does so many amazing things (recursive CTEs, jsonb, etc) that actively make our system better. If there was a fork of Django that optimized for leveraging advanced postgres features, I'd use it.

Sqlite supports recursive CTE

And JSONB, but you can't insert or update in a CTE like you can with Postgres.

Re: Database mocks are not worth it

#94

Post by Deukhoofd »

pdimitar wrote: post #59
Deukhoofd wrote: post #24

Earlier quoted context omitted.

The code being tested also uses transactions internally at times, so it'd mean additional complexity in the code being tested to allow for unit testing, which is not great. In my experience throwing up a database including all tables in an in-memory SQLite db is extremely fast, so it's not really a major concern.

OK, but is your production DB also SQLite? If not, I would not. I found the differences between it and PostgreSQL too big and was getting too many false positives. Also code complexity on a few levels of recursion of transactions is an easy thing to abstract away with almost zero performance penalty -- depending on your programming language of choice.

The production DB is MSSQL, and we scaffold it in SQLite through EF Core. The resulting SQLite DB is close enough to our production DB that we are able to catch invalid defaults, missing foreign keys, etc. in unit tests instead of later on in our testing pipeline, which helps massively in accelerating our development. It could be even better if SQLite would actually tell you which foreign key constraint failed instead of its somewhat unhelpful 'foreign key constraint failed' error, but as it is we at least know something is wrong in our code.

And sure, we could probably refactor it to use transactions to shave a few seconds of running our test suite, but it'd add some additional mental complexity to our codebase to do so. In general, I prefer to keep the actual running code as simple as possible, and have any complexity that is required for unit tests be handled by the unit tests. By just recreating the database the unit tests currently handle all that complexity.

The idea is definitely not a bad one though, and if your scaffolds are big enough to actually cause performance issues with your unit tests it might definitely be a consideration to not recreate the database every time.

Re: Database mocks are not worth it

#95

Post by nrr »

I think it's probably worth mentioning that the principal concern for tests should be proving out the application's logic, and unless you're really leaning on your database to be, e.g., a source of type and invariant enforcement for your data, any sort of database-specific testing can be deferred to integration and UAT.

I use both the mocked and real database approaches illustrated here because they ultimately focus on different things: the mocked approach validates that the model is internally consistent with itself, and the real database approach validates that the same model is externally consistent with the real world.

It may seem like a duplication of effort to do that, but tests are where you really should Write Everything Twice in a world where it's expected that you Don't Repeat Yourself.

Re: Database mocks are not worth it

#96

Post by jaza »

You should do, and I usually do, both "pure" unit tests (with all I/O - database calls, reading/writing local files, 3rd party API calls, etc - being mocked), and integration tests (with ideally all I/O really happening). More of the former, and less of the latter, in line with the "testing pyramid" approach.

There is value in testing "assuming that the database returns 3 rows with Foo IDs and Foo Descriptions, my code should serve a JSON response containing the Foo IDs and the Foo Descriptions concatenated together". And there is also value in testing "when my code executes SELECT id, description FROM foo LIMIT 3, the real database should return 3 rows with Foo IDs and Foo Descriptions". Granted, there's also a big advantage in that the former can run much faster, with zero environment setup / teardown complexity required. But my main point is, different test suites should test different things.

However, if you're unable or unwilling to write two different test suites, and if you're able to easily and reliably have a database available in dev / CI, then ok, I concede, just write integration tests.

Re: Database mocks are not worth it

#97

Post by pdimitar »

Deukhoofd wrote: post #94
pdimitar wrote: post #59

Earlier quoted context omitted.

OK, but is your production DB also SQLite? If not, I would not. I found the differences between it and PostgreSQL too big and was getting too many false positives. Also code complexity on a few levels of recursion of transactions is an easy thing to abstract away with almost zero performance penalty -- depending on your programming language of choice.

The production DB is MSSQL, and we scaffold it in SQLite through EF Core. The resulting SQLite DB is close enough to our production DB that we are able to catch invalid defaults, missing foreign keys, etc. in unit tests instead of later on in our testing pipeline, which helps massively in accelerating our development. It could be even better if SQLite would actually tell you which foreign key constraint failed instea…

Well, if you found a productive workflow then who am I to judge, right?

However, if I was hired into your team tomorrow you'll have to fiercely fight with me over this:

> And sure, we could probably refactor it to use transactions to shave a few seconds of running our test suite, but it'd add some additional mental complexity to our codebase to do so.

Various languages and frameworks demonstrate that abstracting this away and using convenient wrappers is more or less trivial, and definitely a solved problem. And the resulting code is simple; you are (mostly) none the wiser that you are actually using a temporary transaction that would be ultimately rolled back.

...Though that is extremely sensitive to which programming language and DB library you are using, of course.

Your approach works, so keep using it but IMO it's a temporary one and it could stop being fit-for-purpose Soon™. You yourself are already aware of its limitations so just keep an eye out for this moment and be decisive when the time comes to migrate away from it.

My $0.02.

Re: Database mocks are not worth it

#98

Post by pdimitar »

phendrenad2 wrote: post #88
pdimitar wrote: post #63

Earlier quoted context omitted.

PostgreSQL stops many data bugs at the door due to being so strict -- something many programmers start rediscovering is a good thing by choosing stricter programming languages with time. I love SQLite to bits but the test harness I have to put around my apps with it is a separate project in itself.

Somehow this is my first time hearing about that benefit of Postgres! (despite having googled for explanations of why people were switching to it). Being on HN proves more valuable than google once again...

Glad to help!

On a philosophical / meta level it's all quite simple: do your damnedest for the computer to do as much of your work for you as possible, really. Nothing much to it.

Strict technologies slap you hard when you inevitably make a mistake so they do in fact do more of your work for you.

Re: Database mocks are not worth it

#99

Post by thom »

avg_dev wrote: post #31
senex wrote: post #4

Does anyone have experience making tests against real databases fast? I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations. I’d love to hear more from anyone that’s solved for some of these constraints!

Don't know what language or database you use, but check this out: https://github.com/peterldowns/pgtestdb If you happen to use Postgres, the approach is ultimately portable: it uses Pg database templates (also, regarding perf, the author recommends using a ramdisk and turning off fsync on your test DBs; you'll see this in the project readme). But you’ll have to write the code yourself.

This is what I do, it has an overhead of about 10-20ms per test and I’ve had zero flakiness. Absolute no brainier from my point of view.

Re: Database mocks are not worth it

#100

Post by dinosaurdynasty »

senex wrote: post #4

Does anyone have experience making tests against real databases fast? I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations. I’d love to hear more from anyone that’s solved for some of these constraints!

https://eradman.com/ephemeralpg/ (for Postgres) plus making sure your test suite is parallel works wonders.
Post reply on HN