Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to test without mocking

amazingcto.com

Post reply on HN
Print view View original

181–190 of 225 posts

Re: How to test without mocking

#181

Post by hot_gril »

whack wrote: post #88
solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

> If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to y…

Yeah, even if there's no sandbox mode, a separate sandbox account will usually do. Sometimes this catches misuse that would've caused rate-limiting in prod. And if a service makes this hard, maybe you shouldn't use it in prod either.

Re: How to test without mocking

#182

Post by nerder92 »

Everytime I read about this kind of argument against mock is usually due to a misunderstanding on why unit tests with mocking exists in the first place. The underlying assumption is that tests are a quality assurance tool but I think that this is true only for E2E (possibly in production). In outside in TDD unit test are used as a design tool not a QA one, and mocking is a convenient way to quickly do that without the need of implementing the next layer, mock usually don’t replace an implemented service that does IO they implemented a noop that triggers an exception (so they your E2E won’t pass until you implement that)

The problem is in the name, unit test should be called implementation spec or in-code documentation.

Each layer of testing has its roles and serves a different purpose.

Re: How to test without mocking

#183

Post by bunderbunder »

stult wrote: post #177
solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

One of the nice things about the .NET ORM EntityFramework is that you can swap a mocked in-memory database for your prod DB with dependency injection, so without modifying your code at all and theoretically without affecting the behavior of the ORM. Which is to say, you're right, it's about using the right tools. Those tools of course vary by ecosystem and so in some cases mocking the database is in fact the correct…

Probably the single most obnoxious production defect I ever found related to a database would never have made it into production if we had been using a real database instead of a test double. It happened because the test double failed to replicate a key detail in the database's transaction isolation rules.

After figuring it out, I swapped us over to running all the tests that hit the database against the real database, in a testcontainer, with a RAM disk for minimizing query latency. It was about a day's worth of work, and turned up a few other bugs that hadn't bit us in production yet, too. Also sailing past our test suite because the test double failed to accurately replicate the behavior in question.

Total time to run CI went up by about 10 seconds. (For local development you could chop that way down by not starting a fresh server instance for every test run.) Given how many person-hours we spent on diagnosing, resolving, and cleaning up after just that first defect, I estimated the nominally slower non-mocked tests are still a net time saver if amortized over anything less than about 50,000 CI runs, and even then we should probably only count the ones where an engineer is actually blocking on waiting for the tests to complete.

That said, there was a time when I thought test doubles for databases was the most practical option because testing against real databases while maintaining test isolation was an unholy PITA. But that time was 5 or 6 years ago, before I had really learned how to use Docker properly.

Re: How to test without mocking

#184

Post by nerder92 »

postalrat wrote: post #43
whoisthemachine wrote: post #15

> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs. This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.

I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do. Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.

Well tests are usually in English and easier to read then code IMH(uman)O

Re: How to test without mocking

#185

Post by ljm »

whack wrote: post #88
solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

> If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to y…

There are plenty of libraries out there, like VCR, that can set up a test and then save the response for future test runs. You don't really have to renew them that often either.

That was always the go-to for me when testing against 3rd party services, especially because the tests would then survive the offboarding of the engineer who set them up with their personal credentials.

If your test suite relies on live Github PATs or user-specific OAuth access tokens, then you can either figure out how to manage some kind of service account with a 'bot' user, or live with things breaking every time someone leaves the org.

Services that incur a per-request charge, or consume account credits, are another problem. Especially if they don't have sandboxes.

Re: How to test without mocking

#186

Post by d0mine »

thiht wrote: post #161
d0mine wrote: post #146

Earlier quoted context omitted.

Code that uses hexagonal architecture/dependency inversion requires less mocks in their tests.

That’s… not true? No matter how you define your dependencies to inject, if you want to mock the dependencies you inject you have to mock them (it’s almost tautological), no matter if you use dependency inversion or not Maybe you mean "less surface to mock", which is irrelevant if you generate your mocks automatically from the interface

> We can say that a Mock is a kind of spy, a spy is a kind of stub, and a stub is a kind of dummy. But a fake isn’t a kind of any of them. It’s a completely different kind of test double.

You define fakes in this case, not mocks https://blog.cleancoder.com/uncle-bob/2014/05/14/TheLittleMo...

Re: How to test without mocking

#187

Post by [deleted] »

UK-Al05 wrote: post #42
mehagar wrote: post #33

Earlier quoted context omitted.

I agree that if you need to write mocks, it's likely that your interfaces are poorly defined. This is one of the claimed benefits of test driven development - writing the tests first forces you to design the code in a way that cleanly separates modules so they can be tested.

You have to mock/fake when modules call dependencies. Your way means you only ever have siblings. With an orchestrator pulling results out of one module and pushing it into another.

[deleted]

Re: How to test without mocking

#188

Post by dgb23 »

myfavoritetings wrote: post #123
dgb23 wrote: post #89

Earlier quoted context omitted.

Put a small data interface around your IO, have it return DATA | NOT_FOUND etc. Then your tests don't need behavioral mocks or DI, they just need the different shapes of data and you test your own code instead of whatever your IO dependency is or some simulation thereof.

isn't that just mocking with extra steps?

You could call the data you generate for the tests "mocks".

But they really aren't "mocks" in the sense of behavioral mocks via IoC/DI and you don't need to manipulate them via some kind of interface in order to put them into the right state for your particular tests.

There are some extra steps, but you get extremely simple and reliable tests in return.

In many(!) cases you already have a data interface, especially with HTTP/REST APIs. All you need to do is simply not bury the IO call down the stack and maybe describe the failure conditions as plain data in your signature and voila.

(This is not a replacement for higher order testing like, manual, E2E or integration tests. But it certainly beats unit testing with mocks IMO.)

Re: How to test without mocking

#189

Post by curun1r »

Izkata wrote: post #52
postalrat wrote: post #43

Earlier quoted context omitted.

I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do. Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.

Documentation for other developers of that code, not users of that code.

Also worth noting that "other developers" includes the original developer when they return to the code several months later.

Re: How to test without mocking

#190

Post by stult »

bunderbunder wrote: post #183
stult wrote: post #177

Earlier quoted context omitted.

One of the nice things about the .NET ORM EntityFramework is that you can swap a mocked in-memory database for your prod DB with dependency injection, so without modifying your code at all and theoretically without affecting the behavior of the ORM. Which is to say, you're right, it's about using the right tools. Those tools of course vary by ecosystem and so in some cases mocking the database is in fact the correct…

Probably the single most obnoxious production defect I ever found related to a database would never have made it into production if we had been using a real database instead of a test double. It happened because the test double failed to replicate a key detail in the database's transaction isolation rules. After figuring it out, I swapped us over to running all the tests that hit the database against the real databas…

I simply don't think that I will ever be able to come up with anything even vaguely as comprehensive as the test coverage that Microsoft already has for ensuring their ORM behaves consistently across database providers. In my over 10 years of using EF, I have never once encountered a database bug like you describe. If I were to discover such a bug (which I'll admit does occasionally happen even though it hasn't happened to me), it would be easier and better by far to submit an issue to the EF team and let them figure out a fix (including the appropriate tests) than it would be to rework my own test infrastructure. I am not in the business of developing requirements or code for databases, and building an elaborate test model for what I consider the essential requirements for a database would be a distraction from developing code that is more valuable to the business.

The same logic does not apply to all ORMs, of course, which do not all benefit from the same quality of professional dev support that EF receives from MS. But that's my main point from above: the correct design decision depends on the context. For services written in other languages with other ORMs or raw SQL, I absolutely will spin up a full Postgres test container because it is indeed trivial (have one running in the background on my laptop right now in fact). It just isn't necessary in the specific context of EntityFramework code.

Post reply on HN