Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database mocks are not worth it

shayon.dev

Post reply on HN
Print view View original

31–40 of 268 posts

Re: Database mocks are not worth it

#31

Post by avg_dev »

senex wrote: post #4

Does anyone have experience making tests against real databases fast? I resonate with the sentiment of this article, but have struggled to find an alternative that’s fast enough as the test suite grows, isn’t flakey in CI, and is able to share the production schema definition for relevant relations. I’d love to hear more from anyone that’s solved for some of these constraints!

Don't know what language or database you use, but check this out: https://github.com/peterldowns/pgtestdb

If you happen to use Postgres, the approach is ultimately portable: it uses Pg database templates (also, regarding perf, the author recommends using a ramdisk and turning off fsync on your test DBs; you'll see this in the project readme). But you’ll have to write the code yourself.

Re: Database mocks are not worth it

#32

Post by shayonj »

Lvl999Noob wrote: post #26

I faced problems with flaky unit tests that used a common unit test database for the whole project. Since all the tests ran at once, the tests would sometimes fail because of concurrency issues. I never got the time to look too deeply into it. I wonder if there's a database that works sorta like git, giving one "commit" point at the start of every test and then "branching off" so each test can do its own thing, make…

Starting each test in its own transaction and then rolling back at the end, and ensuring no 3rd party network calls in application logic in rails and Postgres apps have worked with a lot of success in my experience.

Re: Database mocks are not worth it

#33

Post by phendrenad2 »

pdimitar wrote: post #17
btilly wrote: post #3

I've found that replacing the database with in memory SQLite for tests is a sweet spot. Almost as fast as a mock, catches a lot of database issues. And it's really easy to do if you're using something like Django that makes automatically generating database migrations easy. It won't help you with database specific differences. But there should be very few of those if you're using a framework that abstracts away the d…

I don't know what kind of magic fairy dust Django is but I've found the differences between SQLite and PostgreSQL too big to be worth it, in at least 3 other programming languages that are not Python. Sounded good at first but we were quickly overwhelmed with false positives and just opted for Postgres in a VM (this was before Docker was a thing).

This is a good thing to be aware of when choosing a database, but I think most of the time people just reach for Postgres because it's the "standard".

Re: Database mocks are not worth it

#34

Post by isotropy »

I've seen a db mock work when 1) there was a small subteam in charge of the OR-mapping, schema structure, and included a DBA; and 2) also a design policy from the architect that all objects had to come out of factories. Under those specific circumstances, having the mock - used solely as a per-developer object cache imitating the factory interface - was critical for unblocking the people working on business logic and front-end.

I wouldn't structure a team that way now.

Re: Database mocks are not worth it

#35

Post by dakiol »

btilly wrote: post #6
javcasas wrote: post #2

For tests, first use a real database. If you can't because the database just doesn't want to (ahem mongo ahem) then use a fake. If you can't use a fake, stop lying and use a fake. Mocks for databases are extremely brittle and complicated.

Ahem MongoDB? I must admit, I've never understood what MongoDB's real use case is supposed to be. Whenever I've looked at it, there has always been a better alternative. When I've had to use it, I've had to debug performance problems that wouldn't have existed with alternative solutions. It sounds cool. But running software isn't about sounding cool. A decade ago, it was really clear. As https://aphyr.com/posts/284-c…

I don’t know. Like any other tool, it depends on how you use it. I worked for a unicorn with presence in multiple countries and they were using Mongo as main db in multiple microservices. Around 1500 engineers. It worked fine. I’m not saying it was justified, but never had perf. issues.

Re: Database mocks are not worth it

#36

Post by whatnotests2 »

Lvl999Noob wrote: post #26

I faced problems with flaky unit tests that used a common unit test database for the whole project. Since all the tests ran at once, the tests would sometimes fail because of concurrency issues. I never got the time to look too deeply into it. I wonder if there's a database that works sorta like git, giving one "commit" point at the start of every test and then "branching off" so each test can do its own thing, make…

Transaction per test is how that works in ruby on rails. Also, the rspec parallel gem will create N test databases for running tests - one per process.

Re: Database mocks are not worth it

#37

Post by masklinn »

Lvl999Noob wrote: post #26

I faced problems with flaky unit tests that used a common unit test database for the whole project. Since all the tests ran at once, the tests would sometimes fail because of concurrency issues. I never got the time to look too deeply into it. I wonder if there's a database that works sorta like git, giving one "commit" point at the start of every test and then "branching off" so each test can do its own thing, make…

Dupe the database. That's what I do in one of my test suites (though that's in part because the tests happen across 4 different processes so it's just not possible to keep it within one transaction).

Creating a db from a template is not free, but if there's not too much stuff in the db it's not horrendous either. And if you're using sqlite, you might be able to backup an on-disk template to an in-memory test db.

Re: Database mocks are not worth it

#38

Post by nerdponx »

btilly wrote: post #6
javcasas wrote: post #2

For tests, first use a real database. If you can't because the database just doesn't want to (ahem mongo ahem) then use a fake. If you can't use a fake, stop lying and use a fake. Mocks for databases are extremely brittle and complicated.

Ahem MongoDB? I must admit, I've never understood what MongoDB's real use case is supposed to be. Whenever I've looked at it, there has always been a better alternative. When I've had to use it, I've had to debug performance problems that wouldn't have existed with alternative solutions. It sounds cool. But running software isn't about sounding cool. A decade ago, it was really clear. As https://aphyr.com/posts/284-c…

I used MongoDB at a company where engineering policy was strictly that MongoDB was our only allowable database, including in cases where it was clearly not the best choice.

Were there good aspects? Sure... kind of. It was super super easy to just throw data into the database. Need to add a field? Who cares, just add it. Need to remove a field? Who cares, just remove it -- as long as the calling code is null-safe. And it was likewise super easy to store denormalized data for fast lookups when it made sense to do so, as well as deeply-nested things and arbitrary JSON blobs synced from our CMS. And queries could be stored and manipulated as actual Python dicts instead of opaque strings, whereas you normally need an ORM or query builder to do that in SQL. And you could get the best-of-both-worlds-ish with the ODMantic framework, where you could spit out a warning (but not an error) and recover gracefully if there happened to be bad data in the database.

Basically it allows you to forego any foresight or serious design, and instead just throw shit together. Which is great for getting a prototype going really fast and giving the appearance of a highly productive team. That is, until you run out of new microservices to prototype and now you have to start adding features to existing code and fixing bugs. IMO it was completely "penny-wise and pound-foolish" with respect to developer time, but I can at least see the appeal if you're a particular type of engineer operating under a particular set of incentives.

As for using MongoDB for things it was actually meant for (writing a ton of schemaless JSON blobs really fast and figuring out reads later), I have no idea because we never used it for that and had nothing really resembling that use case.

Re: Database mocks are not worth it

#39

Post by ariofrio »

rikthevik wrote: post #16
btilly wrote: post #3

I've found that replacing the database with in memory SQLite for tests is a sweet spot. Almost as fast as a mock, catches a lot of database issues. And it's really easy to do if you're using something like Django that makes automatically generating database migrations easy. It won't help you with database specific differences. But there should be very few of those if you're using a framework that abstracts away the d…

> But there should be very few of those if you're using a framework that abstracts away the database. But I really want that database-specific behaviour. :) PostgreSQL does so many amazing things (recursive CTEs, jsonb, etc) that actively make our system better. If there was a fork of Django that optimized for leveraging advanced postgres features, I'd use it.

Does something like PGlite work for your use case? https://pglite.dev/

Re: Database mocks are not worth it

#40

Post by lelandbatey »

btilly wrote: post #6
javcasas wrote: post #2

For tests, first use a real database. If you can't because the database just doesn't want to (ahem mongo ahem) then use a fake. If you can't use a fake, stop lying and use a fake. Mocks for databases are extremely brittle and complicated.

Ahem MongoDB? I must admit, I've never understood what MongoDB's real use case is supposed to be. Whenever I've looked at it, there has always been a better alternative. When I've had to use it, I've had to debug performance problems that wouldn't have existed with alternative solutions. It sounds cool. But running software isn't about sounding cool. A decade ago, it was really clear. As https://aphyr.com/posts/284-c…

I worked full time on a pretty seriously-trafficed product based on MongoDB for 3 years and I still don't know of anywhere I'd want to use MongoDB. I'd basically always want either a DB with a schema or a super fast opaque-store style cache.

Also, their hosted offerings (MongoDB Atlas) were not well operated and they took down our company for 2 days. Our MongoDB instance stopped accepting new connections and restarting our instance via their control panel didn't bring it back up and then their support literally said "we don't know why this happened or how to fix it" for like a day and a half, while we were on their highest tier ultra platinum support contract. Ultimately, we had to do 24 hours of research and then tell their own support how to fix the problem using their special shell access. If I recall correctly, it was some issue with WiredTiger (an internal component of the Mongo version we were using).

After that experience, I'd never use anything produced by MongoDB for anything; we moved everything that we possibly could out of Mongo and into more traditional RDBMS (PostgreSQL) and never had to deal with issues like that again.

Post reply on HN