Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to test without mocking

amazingcto.com

Post reply on HN
Print view View original

81–90 of 225 posts

Re: How to test without mocking

#81

Post by JackFr »

h1fra wrote: post #72
solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

Clearly mocking DB is a footgun and it's not that hard to setup e2e test. Use TestContainer or Docker on a random port, run your API on a random port. Every tests seeds all the data needed to run (user, org, token), it requires an initial setup but then you just reuse it everywhere, and voila. No side effects, no mock to maintain, it also test your auth and permissions, almost 1:1 with prod.

> No side effects, no mock to maintain, it also test your auth and permissions, almost 1:1 with prod.

Can also be used to test version updates of your DB.

Re: How to test without mocking

#82

Post by [deleted] »

Izkata wrote: post #52
postalrat wrote: post #43

Earlier quoted context omitted.

I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do. Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.

Documentation for other developers of that code, not users of that code.

[deleted]

Re: How to test without mocking

#83

Post by postalrat »

randomdata wrote: post #53
postalrat wrote: post #43

Earlier quoted context omitted.

I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do. Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.

A well designed API will generally allow users to understand usage without any additional documentation, sure. However , those who modify the API in the future will want to know every last detail that you knew when you were writing it originally. That must be documented to ensure that they don't get something wrong and break things – and for their general sanity. That is, unless you hate future developers for some re…

Undocumented and undefined are the same thing. If you are worried about changing undefined features then you must be in some sort of legacy hell.

Re: How to test without mocking

#84

Post by lonelyasacloud »

time0ut wrote: post #47

Maybe I am missing something, but how else would I test various exception handling paths? There is a whole world of errors that can occur during IO. What happens if I get a 500 from that web service call? How does my code handle a timeout? What if the file isn't found? It is often only possible to simulate these scenarios using a mock or similar. These are also code paths you really want to understand.

I don't think there's a disagreement; the author states "Whenever I look at mocks, they mostly have the same problem as all unit tests that I see, they only model the happy path". So by corollary their opinion of the correct usage of mocking would also include modelling brokenness.

Re: How to test without mocking

#85

Post by aleksiy123 »

If the normal way of testing is passing some parameters to test some code is what I'm going to call "outside-in" testing.

Then mocking is "inside-out" testing. You check that your code is passing the right params/request to some dependency and reacting correctly to the output/response.

Its really the same thing and you can flip between them by "inverting".

Sometimes mocking just makes much more sense, and sometimes just passing paramaters to a function directly does. The end goal is the same: test some unit of codes behaviour against some specific state/situation.

They have their place but like all testing should be layered with other types of test to "test in depth".

Re: How to test without mocking

#86

Post by ickyforce »

_hl_ wrote: post #17
solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

Re. postgres, this is actually something I have always struggled with, so would love to learn how others do it. I’ve only ever worked in very small teams, where we didn’t really have the resources to maintain nice developer experiences and testing infrastructure. Even just maintaining representative testing data to seed a test DB as schemas (rapidly) evolve has been hard. So how do you - operate this? Do you spin up…

I have tried various approaches and here's what worked best, assuming that there is some natural way to partition most of the data (e.g. per account):

1. Init the DB with some "default" data - configuration, lookup tables, etc

2. Each test in the test suite owns its data. It creates a new account and inserts new records only for that account. It can for example create users on this account, new entities, etc. It can run multiple transactions, can do rollbacks if needed. It is important to only touch the account(s) created by the test and to avoid touching the initial configuration. There's no need to clean up the data after the test finishes. These tests can run concurrently.

3. Create a separate integration test suite which runs sequentially and can do anything with the database. Running sequentially means that these tests can do anything - e.g. test cross-account functionality, global config changes or data migrations. In practice there aren't that many of those, most tests can be scoped to an account. These tests have to clean up after themselves so the next one starts in a good state.

Other approaches had tons of issues. For example if each test is wrapped with a transaction which is later rolled back then testing is very limited - tests cannot use transactions on their own. Savepoints have similar issue.

Re: How to test without mocking

#87

Post by randomdata »

postalrat wrote: post #83
randomdata wrote: post #53

Earlier quoted context omitted.

A well designed API will generally allow users to understand usage without any additional documentation, sure. However , those who modify the API in the future will want to know every last detail that you knew when you were writing it originally. That must be documented to ensure that they don't get something wrong and break things – and for their general sanity. That is, unless you hate future developers for some re…

Undocumented and undefined are the same thing. If you are worried about changing undefined features then you must be in some sort of legacy hell.

Nothing is left undefined. Absent of documentation, most likely something will end up defined by inference. Which is not a good place to be as a developer as you have lost the nuance that went into it originally.

Re: How to test without mocking

#88

Post by whack »

solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

> If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite.

I've been doing E2E testing using 3rd-party APIs for a decade now, and this has yet to be a significant problem. The majority of my APIs had a dedicated sandbox environment to avoid "rate limiting, bans, and other anti-abuse mechanisms". The remainder were simple enough that the provider didn't care about users exploring on the live API, and were usually read-only as well.

Did I run into the occasional flaky failure, or API stability issues? Sure. But it was very rare and easy to workaround. It never devolved into becoming "untestable" or "full manual QA"

My other teams that relied on mocks suffered from far worse problems - a ton of time being spent on manual-QA, and bugs that leaked into production, because of mock-reality mismatches.

Re: How to test without mocking

#89

Post by dgb23 »

time0ut wrote: post #47

Maybe I am missing something, but how else would I test various exception handling paths? There is a whole world of errors that can occur during IO. What happens if I get a 500 from that web service call? How does my code handle a timeout? What if the file isn't found? It is often only possible to simulate these scenarios using a mock or similar. These are also code paths you really want to understand.

Put a small data interface around your IO, have it return DATA | NOT_FOUND etc.

Then your tests don't need behavioral mocks or DI, they just need the different shapes of data and you test your own code instead of whatever your IO dependency is or some simulation thereof.

Re: How to test without mocking

#90

Post by senordevnyc »

chuckadams wrote: post #54
whoisthemachine wrote: post #15

> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs. This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.

> Unit tests aren't meant to find bugs, they're meant to protect against regressions That hasn't been the general consensus on unit tests for at least 30 years now. Regression tests are a small subset of tests, typically named for an ID in some bug tracker, and are about validating a fix. The majority of unit tests catch issues before a bug is even opened, and pretty much any random developer you talk to will conside…

The majority of unit tests catch issues before a bug is even opened

The "issue" that is being caught is the bug the parent is talking about, not a "bug" in JIRA or something.

Post reply on HN