How to test without mocking • Hacker News Board

Re: How to test without mocking

#171

Post by remixff2400 » Mon, Jun 17, 2024, 8:19 PM UTC

mvdtnz wrote: post #153

CuriouslyC wrote: post #110

Earlier quoted context omitted.

I've worked with a lot of pro-mocking engineers, and the time they spent on mocks easily outstripped the time a good build engineer would have spent creating a fast reusable test framework using real databases/dummy services/etc. The mocks won not because they were better or more efficient, but because of lack of deeper engineering skill and cargo culting.

That engineer may have spent a couple of dozen hours on their mock. But the engineers who spent time on a test framework that uses real databases will soak up thousands of developer hours in CI time over the next decade.

Spinning up DB instances is a lot faster now than it used to be in the past. There are even modules for in-memory instances of certain databases. The speed of a unit test vs. that of one that uses an actual database is small enough for it to be a real consideration.

That being said, of course, "it depends" on your use case. But I've found setting up this sort of test environment quite a bit easier now than writing database mocks, a lot less time-and-maintenance intensive, and relatively quick to run in any environment.

(Also, in a decade, I'm pretty confident this gap will get even smaller, while the time to maintain mocks will stay constant)

Re: How to test without mocking

#172

Post by alexfromapex » Mon, Jun 17, 2024, 8:21 PM UTC

It is so cringe to see bad advice like this being given. Yes, you can write mocks incorrectly. You should not model them after the "happy path" but you should make sure they cover the most use-cases both good and bad. I have been a senior or principal engineer on teams that did both of these approaches and the non-mocking approach is terrible because you end up with separate tests that have colliding data. It's slower using a real database back-end and becomes a mess and leads to issues where your database is heavily coupled to your test code which is the real anti-pattern. Then a year or two later when you want to change databases or database architectures you're screwed because you have to go into a bunch of tests manually and change things. The whole point of the mocks is it makes everything modular.

Re: How to test without mocking

#173

Post by rkeene2 » Mon, Jun 17, 2024, 8:23 PM UTC

RonnieOwnsLexus wrote: post #135

i dont get it. I if am taking a dependency on database or another class and i mock it using its interface, what is the harm in it? Essentially i have tested that given my dependencies working correctly my class would also work as expected.

Almost -- you're testing that given your mocking implementation perfectly mirrors what the dependency would do given the inputs tested with that your functions produce the correct outputs (and hopefully you also verified the side-effects).

The article is stating that almost nobody goes through the trouble of implementing a mock database perfectly, they just do something like make a single call return some hard-coded data. While this works a bit, it means that if the database ever changes its interface you have to remember to notice and implement that change as well.

Re: How to test without mocking

#174

Post by CuriouslyC » Mon, Jun 17, 2024, 8:29 PM UTC

brushfoot wrote: post #167

CuriouslyC wrote: post #110

Earlier quoted context omitted.

I've worked with a lot of pro-mocking engineers, and the time they spent on mocks easily outstripped the time a good build engineer would have spent creating a fast reusable test framework using real databases/dummy services/etc. The mocks won not because they were better or more efficient, but because of lack of deeper engineering skill and cargo culting.

> the time they spent on mocks easily outstripped the time a good build engineer would have spent creating a fast reusable test framework This goes back to team size and skills. Not all teams have build engineers. And not all mocks are so complicated that they take up that much time. Again, it depends on the scope and the resources. The article goes too far by calling mocking an anti-pattern. It simply isn't.

If I didn't have the team to properly do a good test build I would tell all my engineers to design their code to not need mocks if at all possible and call it a day at 85% coverage. That's very doable with dependency injection and modular function libraries. The time spent not chasing full coverage could be better spent improving CI/integration testing/deployment.

Re: How to test without mocking

#175

Post by gchaincl » Mon, Jun 17, 2024, 8:36 PM UTC

> This often happens with an in-memory database, which is compatible to your main database you do use in production, but is faster.

Not sure how this will solve edge cases problems described at the beginning of the article

Re: How to test without mocking

#176

Post by lye » Mon, Jun 17, 2024, 8:37 PM UTC

_hl_ wrote: post #17

solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

Re. postgres, this is actually something I have always struggled with, so would love to learn how others do it. I’ve only ever worked in very small teams, where we didn’t really have the resources to maintain nice developer experiences and testing infrastructure. Even just maintaining representative testing data to seed a test DB as schemas (rapidly) evolve has been hard. So how do you - operate this? Do you spin up…

I run a replicated copy of the production database on top of zfs and snapshot it before starting tests. PostgreSQL takes a few seconds to start on the snapshot and then you're off to the races with real production data. When the test suite finishes, the snapshot is discarded. This also ensures that migrations apply correctly to the production db before an actual prod is used.

Re: How to test without mocking

#177

Post by stult » Mon, Jun 17, 2024, 8:41 PM UTC

solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

One of the nice things about the .NET ORM EntityFramework is that you can swap a mocked in-memory database for your prod DB with dependency injection, so without modifying your code at all and theoretically without affecting the behavior of the ORM. Which is to say, you're right, it's about using the right tools. Those tools of course vary by ecosystem and so in some cases mocking the database is in fact the correct decision.

Re: How to test without mocking

#178

Post by brushfoot » Mon, Jun 17, 2024, 8:41 PM UTC

CuriouslyC wrote: post #174

brushfoot wrote: post #167

Earlier quoted context omitted.

> the time they spent on mocks easily outstripped the time a good build engineer would have spent creating a fast reusable test framework This goes back to team size and skills. Not all teams have build engineers. And not all mocks are so complicated that they take up that much time. Again, it depends on the scope and the resources. The article goes too far by calling mocking an anti-pattern. It simply isn't.

If I didn't have the team to properly do a good test build I would tell all my engineers to design their code to not need mocks if at all possible and call it a day at 85% coverage. That's very doable with dependency injection and modular function libraries. The time spent not chasing full coverage could be better spent improving CI/integration testing/deployment.

It depends. The type of project affects the decision too.

If it's a small, standalone CRUD app on a SQLite database, mocks would probably be a bad option, sure.

On the other hand, it could be an integration platform that integrates with many third-party services. Some of them may not have test environments. Or some of the integrations may be written by third-party contractors, and we can't expose service credentials because of poor permissions granularity. Mocks are a good option there.

Re: How to test without mocking

#179

Post by hot_gril » Mon, Jun 17, 2024, 8:53 PM UTC

0xbadcafebee wrote: post #154

solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

> If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, Actually that's wrong too. The production database will be different than the "testing Postgres instance", leading to bugs. It turns out that whatever testing solution you use, if it's not the actual production instance and you're not using real production data, there will be b…

This doesn't mean the solution of testing with a separate Postgres instance is wrong.

Re: How to test without mocking

#180

Post by hot_gril » Mon, Jun 17, 2024, 8:54 PM UTC

jiehong wrote: post #111

_hl_ wrote: post #17

Earlier quoted context omitted.

Re. postgres, this is actually something I have always struggled with, so would love to learn how others do it. I’ve only ever worked in very small teams, where we didn’t really have the resources to maintain nice developer experiences and testing infrastructure. Even just maintaining representative testing data to seed a test DB as schemas (rapidly) evolve has been hard. So how do you - operate this? Do you spin up…

Great answers below (test containers for example). However, it’s not always possible. For example: - you use oracle db (takes minutes to start, license, hope the containers run on ARM fine, etc.) - sometimes an in memory fake is just much faster, and can be an official db on its own for people to try the product - your storage might be only available through a library by a third party provider that is not available l…

Could you leave a test Oracle DB running all the time and clear it between tests? I do this with Postgres.