Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why a mock doesn’t work

nedbatchelder.com

Post reply on HN
Print view View original

21–30 of 77 posts

Re: Why a mock doesn’t work

#21

Post by PopeDotNinja »

hinkley wrote: post #16
stephen wrote: post #13

Agreed with the articles, mocks are very tied to implementation details. I almost always prefer state-based testing: https://martinfowler.com/articles/mocksArentStubs.html As "observable state goes from a to b" is much closer to a business/functional requirement that will still/always be true, regardless of refactorings. Refactoring in codebases with state-based tests is a pleasure; in codebases with mock-based tests…

Dependency injection leads to elaborate, brittle fixtures that become more elaborate and arcane over time. And the sunk cost fallacy has people working for hours (and I’ve witnessed pairs spend two days, that’s over three man days!) trying to maintain them. I use mocks to arrange state A without going through all the business rules involved in creating state A. But you have to expose the states to do it, which can ma…

> The absolute worst is fixtures with asynchronous code

I concur that setting up a test for async code is a complete pain in the arse. It's extra fun when the caller doesn't get a reply, so there's no response that you can assert your as valid or invalid. It's extra, extra fun when the tests in your test suite need to run in parallel without stepping on each other.

Here's an example. Say you've got PipelineService123 that receives a message with a chunk of data, transforms that data, and sends it elsewhere. Try thinking about dynamically setting an instance of PipelineService123, wiring up an input message sender, giving that sender the dynamic address of the instance of PipelineService123, wiring up an out message receiver, making sure the instance of PipelineService123 knows how to reach the receiver, all to test whether or not everything is wired up correctly & that the data is being sent, transformed, and received properly, running this test in parallel with all of the other test cases, and keeping it reasonably easy to reason about. Good luck with that! I'm not sure even I even stayed on top of my attempt to describe it :P

Re: Why a mock doesn’t work

#22

Post by askvictor »

I've always been dubious of using mock for tests which involve external API calls; at best they require you to reimplement the API according to the documentation (which there may be none, or that the API doesn't follow exactly for edge cases (i.e. the things you're meant to be testing)). At worst you're implementing a very small subset of the behaviour of the API and not testing how your code responds to the other behaviour. But I haven't come across other solutions (not that I have that a heap of experience here, just contributions to a couple of oss projects).

Re: Why a mock doesn’t work

#23

Post by alexanderdmitri »

askvictor wrote: post #22

I've always been dubious of using mock for tests which involve external API calls; at best they require you to reimplement the API according to the documentation (which there may be none, or that the API doesn't follow exactly for edge cases (i.e. the things you're meant to be testing)). At worst you're implementing a very small subset of the behaviour of the API and not testing how your code responds to the other be…

Mocking inputs from APIs is actually great if someone besides the original developer picks up the codebase, whether from a system or even down to isolated function stubs. This is because it gives insight into the expected inputs the original developer(s) had been expecting and can elucidate the source under test.

Integration tests fill the hole you're pointing out. You can even have integration tests designed to validate the mock inputs in many cases.

Re: Why a mock doesn’t work

#24

Post by kdazzle »

zzzeek wrote: post #3
llarsson wrote: post #2

For a person who does not code very large programs in Python, this looks scary and like something that is hardly "only one way to do it" and that the most elegant solution gives you what you want. How I import things (or the libraries I depend on!) affects how I can write my tests? Really? My own Python scripts are typically single-screen in length. And one-off stuff, where they either work or don't, basically. Is th…

no, this blog post is absolutely on point and correct. it can be extremely inconvenient and complicated to get around using mocks in many situations where you want to test things, so we want to use mocks when appropriate. Then, you definitely want to mock at the most specific level possible. while mocking in a way that is specific to how modules are imported is technically "fragile", in that it is deeply dependent on…

Yes - totally agree. Sure, mocking everything is a bad idea. And mocking can make changes brittle. But I would much rather just mock HTTP requests, dates, and filesystem stuff than spend a day figuring out how to write some wrapper around HTTP requests to inject in .NET core.

Re: Why a mock doesn’t work

#26

Post by hinkley »

PopeDotNinja wrote: post #21
hinkley wrote: post #16

Earlier quoted context omitted.

Dependency injection leads to elaborate, brittle fixtures that become more elaborate and arcane over time. And the sunk cost fallacy has people working for hours (and I’ve witnessed pairs spend two days, that’s over three man days!) trying to maintain them. I use mocks to arrange state A without going through all the business rules involved in creating state A. But you have to expose the states to do it, which can ma…

> The absolute worst is fixtures with asynchronous code I concur that setting up a test for async code is a complete pain in the arse. It's extra fun when the caller doesn't get a reply, so there's no response that you can assert your as valid or invalid. It's extra, extra fun when the tests in your test suite need to run in parallel without stepping on each other. Here's an example. Say you've got PipelineService123…

I've noticed that my tests are a lot less tortured when the language supports async/await.

But I do wonder if await should be the default behavior, since most async code is still sequential within the operation, at least until we start composing async operations. In which case having the keywords point out the composition probably improves reading comprehension.

Re: Why a mock doesn’t work

#27

Post by raymondh »

I think Ned strikes the right balance, showing risks associated with mock objects without condemning them outright. There is no doubt that people sometimes go overboard with mocking and there is no doubt that there are situations where it is really helpful.

Re: Why a mock doesn’t work

#28

Post by frenchman99 »

mpweiher wrote: post #17

Why I don't mock: https://blog.metaobject.com/2014/05/why-i-don-mock.html

You're talking about mocking database calls though. In my line of work (insurance brokerage), we use lots of insurance APIs and they are sometimes very slow (+20 seconds / call) or completely down at random hours. There is simply no way around mocking those API calls if you want a fast and reliable testsuite.

Re: Why a mock doesn’t work

#29

Post by switchbak »

squirrelicus wrote: post #14
stephen wrote: post #13

Agreed with the articles, mocks are very tied to implementation details. I almost always prefer state-based testing: https://martinfowler.com/articles/mocksArentStubs.html As "observable state goes from a to b" is much closer to a business/functional requirement that will still/always be true, regardless of refactorings. Refactoring in codebases with state-based tests is a pleasure; in codebases with mock-based tests…

Even better than "observable state from a to b" is "a transform of value from a to b", no inspection and whitebox needed :D

Absolutely. The more functional we can make the code, typically the easier it is to test it based only on inputs and outputs.

Much of what I do is try to come up with architectures that model the logic in this way, and are still legible to my co workers.

Re: Why a mock doesn’t work

#30

Post by jabwork »

I've wrestled with this problem several times. My conclusion was mocks are just fine, but this is a wart in that there isn't "one way to do it"

For the most part I can get by with one rule: always mock the module.

  with mock.patch('os.listdir'):
will always work, even if it doesn't accomplish what you want.

  with mock.patch('mymodule.os.listdir')
will fail if that module does not explicitly import os and instead does something like from os import listdir (perhaps because a later dev did not realize importing os directly was actually a requirement for the test and changed the code).

The rule is not perfect though. In the above case, the error will actually be

  ImportError: No module named os
This can be fixed with e.g.,

  assert hasattr(module, 'os'), "os module is not explicitly imported"
as a preamble to your test but ... it is not perfect by any means.

EDIT: formatting

Post reply on HN