Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to test without mocking

amazingcto.com

Post reply on HN
Print view View original

191–200 of 225 posts

Re: How to test without mocking

#191

Post by time0ut »

dgb23 wrote: post #89
time0ut wrote: post #47

Maybe I am missing something, but how else would I test various exception handling paths? There is a whole world of errors that can occur during IO. What happens if I get a 500 from that web service call? How does my code handle a timeout? What if the file isn't found? It is often only possible to simulate these scenarios using a mock or similar. These are also code paths you really want to understand.

Put a small data interface around your IO, have it return DATA | NOT_FOUND etc. Then your tests don't need behavioral mocks or DI, they just need the different shapes of data and you test your own code instead of whatever your IO dependency is or some simulation thereof.

Sure. This is a good practice for multiple reasons. However, the code that glues my interface to the underlying I/O is still there and needs testing, right?

I agree with you in general. But it always feels like there are spots where a mock of some kind is the only way to cover certain things.

Re: How to test without mocking

#192

Post by smackeyacky »

stult wrote: post #190
bunderbunder wrote: post #183

Earlier quoted context omitted.

Probably the single most obnoxious production defect I ever found related to a database would never have made it into production if we had been using a real database instead of a test double. It happened because the test double failed to replicate a key detail in the database's transaction isolation rules. After figuring it out, I swapped us over to running all the tests that hit the database against the real databas…

I simply don't think that I will ever be able to come up with anything even vaguely as comprehensive as the test coverage that Microsoft already has for ensuring their ORM behaves consistently across database providers. In my over 10 years of using EF, I have never once encountered a database bug like you describe. If I were to discover such a bug (which I'll admit does occasionally happen even though it hasn't happe…

I can think of a few things that will upset this particular apple cart, chief amongst them is the behaviour of different databases and sorting / collation which might not be generally categorised as the kind of bug a test suite will uncover, but certainly creates production bugs / issues.

I love EntityFramework, it's easily the best ORM I have ever used but it has a few cross-platform footguns that require testing against the actual database service you're using.

Re: How to test without mocking

#193

Post by 0xbadcafebee »

thiht wrote: post #159
0xbadcafebee wrote: post #154

Earlier quoted context omitted.

> If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, Actually that's wrong too. The production database will be different than the "testing Postgres instance", leading to bugs. It turns out that whatever testing solution you use, if it's not the actual production instance and you're not using real production data, there will be b…

> The production database will be different than the "testing Postgres instance", leading to bugs. It never happened to me to be honest. This reads an argument for "if you can’t do perfect, just do it badly" but it’s nonsense. Running tests against a local Postgres instance with the same major.minor version and same extensions as your prod instance WILL work. And testing your storage layer against the database is pro…

> Running tests against a local Postgres instance with the same major.minor version and same extensions as your prod instance WILL work.

A team I worked with recently said the same thing. But, as I predicted, they ran into bugs because the CloudSQL Postgres was different than their Dockerized Postgres, even though it was the same core version.

There will always be testing problems you can't anticipate. Especially with systems that are not your own code. Just be ready to adapt your testing when it doesn't work as expected, and don't invest too much in the testing if it's not worth it.

Re: How to test without mocking

#194

Post by whoisthemachine »

chuckadams wrote: post #54
whoisthemachine wrote: post #15

> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs. This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.

> Unit tests aren't meant to find bugs, they're meant to protect against regressions That hasn't been the general consensus on unit tests for at least 30 years now. Regression tests are a small subset of tests, typically named for an ID in some bug tracker, and are about validating a fix. The majority of unit tests catch issues before a bug is even opened, and pretty much any random developer you talk to will conside…

They do not "find" bugs in the way that exploratory testing or user operation might (or even in the way that broader integration tests might), that is they don't find bugs that are not in the known problem space. But they are very good at proving a method works correctly and covers the known execution permutations.

Re: How to test without mocking

#195

Post by alexfromapex »

smackeyacky wrote: post #192
stult wrote: post #190

Earlier quoted context omitted.

I simply don't think that I will ever be able to come up with anything even vaguely as comprehensive as the test coverage that Microsoft already has for ensuring their ORM behaves consistently across database providers. In my over 10 years of using EF, I have never once encountered a database bug like you describe. If I were to discover such a bug (which I'll admit does occasionally happen even though it hasn't happe…

I can think of a few things that will upset this particular apple cart, chief amongst them is the behaviour of different databases and sorting / collation which might not be generally categorised as the kind of bug a test suite will uncover, but certainly creates production bugs / issues. I love EntityFramework, it's easily the best ORM I have ever used but it has a few cross-platform footguns that require testing ag…

If you need to test collation or some database implementation detail, that is the exception where it may be good to use a real database but certainly is not the rule.

Re: How to test without mocking

#196

Post by Izkata »

chuckadams wrote: post #54
whoisthemachine wrote: post #15

> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs. This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.

> Unit tests aren't meant to find bugs, they're meant to protect against regressions That hasn't been the general consensus on unit tests for at least 30 years now. Regression tests are a small subset of tests, typically named for an ID in some bug tracker, and are about validating a fix. The majority of unit tests catch issues before a bug is even opened, and pretty much any random developer you talk to will conside…

> Regression tests are a small subset of tests, typically named for an ID in some bug tracker, and are about validating a fix.

This is how I also tend to think of them, but it's not how the phrase is generally used. The general meaning of regression tests it to ensure known correct functionality doesn't break with a future change. There's no actual requirement it be tied to a known bug.

Re: How to test without mocking

#197

Post by Foobar8568 »

alexfromapex wrote: post #195
smackeyacky wrote: post #192

Earlier quoted context omitted.

I can think of a few things that will upset this particular apple cart, chief amongst them is the behaviour of different databases and sorting / collation which might not be generally categorised as the kind of bug a test suite will uncover, but certainly creates production bugs / issues. I love EntityFramework, it's easily the best ORM I have ever used but it has a few cross-platform footguns that require testing ag…

If you need to test collation or some database implementation detail, that is the exception where it may be good to use a real database but certainly is not the rule.

Collation, localization ( Switzerland is a mess on .Net, I don't want to imagine other frameworks) etc.

I ended up a few times with discrepancy in the format (excel, .net and windows ) because someone changed it.

Re: How to test without mocking

#198

Post by Too »

switchbak wrote: post #112
chii wrote: post #107

Earlier quoted context omitted.

> because of mock-reality mismatches. you also need to test the mocks against the real thing separately.

I would prefer that we all try to use this language consistently: https://www.martinfowler.com/articles/mocksArentStubs.html What you're describing sounds like a fake to me.

Good luck with that. Any org I’ve seen can’t even consistently agree what is a unit test, component test, black box test vs integration test.

Re: How to test without mocking

#199

Post by thiht »

d0mine wrote: post #186
thiht wrote: post #161

Earlier quoted context omitted.

That’s… not true? No matter how you define your dependencies to inject, if you want to mock the dependencies you inject you have to mock them (it’s almost tautological), no matter if you use dependency inversion or not Maybe you mean "less surface to mock", which is irrelevant if you generate your mocks automatically from the interface

> We can say that a Mock is a kind of spy, a spy is a kind of stub, and a stub is a kind of dummy. But a fake isn’t a kind of any of them. It’s a completely different kind of test double. You define fakes in this case, not mocks https://blog.cleancoder.com/uncle-bob/2014/05/14/TheLittleMo...

If you want. Replace mock by fake in my post and you get the same thing. It means I’m using fakes too when I’m not doing dependency inversion, so no mocks either

For what it’s worth, I find the distinction between mocks, fakes, spies, stubs, dummies and whatever completely useless in practice, the whole point is to control some data flows to test in isolation, it’s really all that matters.

Fun thing this kind of bikeshedding comes from Bob Martin, who famously has nothing worthwhile to show as an example of his actual work. Every time I read something from him, I get more and more convinced he’s a total fraud.

Re: How to test without mocking

#200

Post by thiht »

0xbadcafebee wrote: post #193
thiht wrote: post #159

Earlier quoted context omitted.

> The production database will be different than the "testing Postgres instance", leading to bugs. It never happened to me to be honest. This reads an argument for "if you can’t do perfect, just do it badly" but it’s nonsense. Running tests against a local Postgres instance with the same major.minor version and same extensions as your prod instance WILL work. And testing your storage layer against the database is pro…

> Running tests against a local Postgres instance with the same major.minor version and same extensions as your prod instance WILL work. A team I worked with recently said the same thing. But, as I predicted, they ran into bugs because the CloudSQL Postgres was different than their Dockerized Postgres, even though it was the same core version. There will always be testing problems you can't anticipate. Especially wit…

Which bugs?

I’ve been ready for 5 years as it’s the duration I’ve been testing my storage layers with 90% integration tests (the 10% being the hard to reproduce error cases, these tests have low value but are easy to test with mocks so I still test them). The only issue I’ve encountered was with time zones (shocking), and it made me ensure I got full control of the time zones in my app, in my deployment, in my local Postgres and in my prod Postgres, so net benefit in the end.

Post reply on HN