Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to test without mocking

amazingcto.com

Post reply on HN
Print view View original

211–220 of 225 posts

Re: How to test without mocking

#211

Post by projektfu »

choeger wrote: post #134

A radical point of view. And as such it is of course wrong ;). First of all, there are languages where dry-running your code with all parameters mocked is still a valid test run. Python, js, and Perl for instance make it very simple to have a stupid error in the routine that crashes every run. But more importantly, a unit test usually executes inside the same process as the code. That gives you tremendous introspecti…

I can foresee the "not what I mean" answers to everything. Oh, sure it's a fake DB but that's not a mock. Oh, yeah, you need to test with something that always makes an error but that's not a mock.

Eventually, what they mean is that if it sucks, it's what they're talking about, and you should never do that. If it was really useful, it's not a mock.

Re: How to test without mocking

#212

Post by nickkell »

bunderbunder wrote: post #183
stult wrote: post #177

Earlier quoted context omitted.

One of the nice things about the .NET ORM EntityFramework is that you can swap a mocked in-memory database for your prod DB with dependency injection, so without modifying your code at all and theoretically without affecting the behavior of the ORM. Which is to say, you're right, it's about using the right tools. Those tools of course vary by ecosystem and so in some cases mocking the database is in fact the correct…

Probably the single most obnoxious production defect I ever found related to a database would never have made it into production if we had been using a real database instead of a test double. It happened because the test double failed to replicate a key detail in the database's transaction isolation rules. After figuring it out, I swapped us over to running all the tests that hit the database against the real databas…

Using an in-memory database does not increase my confidence that in my software either. I also started using dockerised dependencies in tests a couple of years ago.

Can you please explain what you did with a RAM disk to speed them up?

Re: How to test without mocking

#213

Post by yellowapple »

solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

> Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite.

Those rate limits, bans, and other anti-abuse mechanisms are things that would be good to uncover and account for during tests. Better for the test suite to detect those potential failures than the production deployment :)

Re: How to test without mocking

#214

Post by lelanthran »

JasonSage wrote: post #210
lelanthran wrote: post #202

Earlier quoted context omitted.

> Every place I've ever worked which tried this has managed to get a production database deleted by somebody running tests. That's just a different way of saying "Every place I've ever worked at use production for testing" :-/ TBH, any place using the same credentials for test and for production have bigger problems than would ever be fixed by mocking.

You've made multiple assumptions here that couldn't be further from reality. You don't have to use production for testing or share test credentials between environments for automated tests to exploit an unintended environment, you just have to forget to clean up a `.env` file after testing a hotfix. ... is that good? Hell no. But it's a much more common version of reality than you're assuming is happening.

> You've made multiple assumptions here that couldn't be further from reality.

Sure I've made an assumption, but in my defence it's a single reasonable assumption: that one wouldn't be running tests in production.

If you have a leftover `.env` that contains production credentials, you were running your test in production.

Re: How to test without mocking

#215

Post by Olreich »

bunderbunder wrote: post #132

I used to love mocks, once upon a time. Nowadays, though, I internally sigh when I see them. I've come to the opinion that test doubles of any kind should be used as a last resort. They're a very useful tool for hacking testability into legacy code that's not particularly testable. But in a newer codebase they should be treated as a code smell. Code that needs mocks to be tested tends to be code that is overly statef…

At some point, you need to interact with something that looks like I/O or an external service. Not handling failures from them is a source of a lot of bugs.

Even if pushed to the periphery, how do you test the wrapper you built to hide these failures from the rest of your code base? If you don’t hide these failures in some wrapper, how do you test that your system handles them properly?

Re: How to test without mocking

#216

Post by JasonSage »

lelanthran wrote: post #214
JasonSage wrote: post #210

Earlier quoted context omitted.

You've made multiple assumptions here that couldn't be further from reality. You don't have to use production for testing or share test credentials between environments for automated tests to exploit an unintended environment, you just have to forget to clean up a `.env` file after testing a hotfix. ... is that good? Hell no. But it's a much more common version of reality than you're assuming is happening.

> You've made multiple assumptions here that couldn't be further from reality. Sure I've made an assumption, but in my defence it's a single reasonable assumption: that one wouldn't be running tests in production. If you have a leftover `.env` that contains production credentials, you were running your test in production .

What you’re saying is tautological: your assumption is that tests were running in production, and your evidence is that somebody accidentally ran tests in production. It reads a bit like “Oh my, an accident? Why’ve you done that.”

Anyway, the missed point is that you can’t just do anything in tests and just expect the best of intentions to ensure it doesn’t backfire. One must also consider the security situation, infrastructure, secrets management, shared environments, and more. It’s not as simple as just plopping down a test database and expecting everything to go smoothly. You wouldn’t be that careless with other things, so don’t do it with tests, and don’t rely on “don’t run tests in production” as your only safeguard.

Re: How to test without mocking

#217

Post by bottlepalm »

t43562 wrote: post #69
bottlepalm wrote: post #58

Earlier quoted context omitted.

Also the intermittent failures of your tests relying on unstable dependencies.

If your dependencies are unstable then that is very important to know! If it means you have to add forms of resilience then that's good for your code perhaps?

The point is it makes you tests nondeterministic which is a huge problem with large fluid teams and thousands of tests. They need to be deterministic.

Re: How to test without mocking

#218

Post by bunderbunder »

nickkell wrote: post #212
bunderbunder wrote: post #183

Earlier quoted context omitted.

Probably the single most obnoxious production defect I ever found related to a database would never have made it into production if we had been using a real database instead of a test double. It happened because the test double failed to replicate a key detail in the database's transaction isolation rules. After figuring it out, I swapped us over to running all the tests that hit the database against the real databas…

Using an in-memory database does not increase my confidence that in my software either. I also started using dockerised dependencies in tests a couple of years ago. Can you please explain what you did with a RAM disk to speed them up?

Configure the container with a tmpfs volume (https://docs.docker.com/storage/tmpfs/), and configure the database server to store all its files in that volume's directory.

Re: How to test without mocking

#219

Post by Fire-Dragon-DoL »

solatic wrote: post #12

If you're writing a CRUD app and mocking your database calls instead of just starting an actual Postgres instance before running the tests, you're probably using mocking wrong. If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at…

100% agree

Re: How to test without mocking

#220

Post by hot_gril »

0xbadcafebee wrote: post #208
thiht wrote: post #200

Earlier quoted context omitted.

Which bugs? I’ve been ready for 5 years as it’s the duration I’ve been testing my storage layers with 90% integration tests (the 10% being the hard to reproduce error cases, these tests have low value but are easy to test with mocks so I still test them). The only issue I’ve encountered was with time zones (shocking), and it made me ensure I got full control of the time zones in my app, in my deployment, in my local…

I don't remember them all off the top of my head, but I do remember: - Access to different parts of the database are limited in CloudSQL since it's a managed database. This makes some features of automated tooling (like migrations) not work on CloudSQL (i'm not saying migrations don't work, i'm saying some features do work, some don't). Sometimes elevated permissions can fix it, but some aspects are just walled off.…

"You only get Read Committed transaction isolation." – That's unexpected, I wonder what within CloudSQL makes the other modes not usable.

If my prod were using CloudSQL, I'd use CloudSQL for tests too. Haven't noticed so many differences between Heroku Postgres and stock.

Post reply on HN