Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When to Mock

enterprisecraftsmanship.com

Post reply on HN
Print view View original

71–80 of 82 posts

Re: When to Mock

#71

Post by mumblemumble »

monkpit wrote: post #47
commandlinefan wrote: post #22

Earlier quoted context omitted.

> scads of bugs that weren't caught by the unit tests Which is fine! Unit tests will never catch all the bugs - neither will type safety. Neither will code reviews. Neither will manual testing. But unit tests do catch the kinds of bugs that unit tests are good at catching, which eases the burden on the manual testers.

> because the test doubles for the database don't accurately emulate important behaviors and semantics of the real database. Commenter implies that bugs occurred in code that was assumed to be tested and correct (according to the test specs) because it did have tests. Which is decidedly Not Fine. Now, would I consider them to be “unit tests” in this case? Probably not. But the label you decide to slap on the test doe…

Whether, in practice, I would consider them to be unit tests probably depends on what my colleagues want to call them, and little else.

Personally, I do prefer a more classicist definition, because I believe that's the more pragmatic one. But I also believe that arguing over the definition of the term "Unit test" is one of the most wasteful possible examples of bikeshedding. Like you imply, the only thing that really matters is the extent to which your test suite gives you confidence that the software behaves correctly.

Re: When to Mock

#72

Post by grey-area »

amw-zero wrote: post #51
grey-area wrote: post #7

Earlier quoted context omitted.

When you're developing, aren't you running a db server with a dev database for that? Just add another separate test db and use that for tests where appropriate. Also, most unit tests don't need a db, it's more useful for integration tests.

Hitting a database in tests is an order of magnitude slower and also involves worrying about a lot more code, i.e. the database engine. You may or may not want to take on that burden.

I use the db for some tests, it's totally fine, because it's limited to those tests which actually need db access (most don't). It's also closer to the actual application behaviour than mocks and doesn't need to be maintained.

Re: When to Mock

#73

Post by regularfry »

mabbo wrote: post #3

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…

I find it utterly bizarre that this should still be a current topic of contention. Running unit tests against a local database was a norm in Rails a decade ago, because the framework made it trivial to set up.

Yes, there were knock-on performance questions that wanted answering - how do you avoid the database setup costs for those tests that don't care about it, for instance, the "fast Rails test" movement was a big thing - but by and large those were solved problems by the time I stopped writing Rails code professionally around the 5.1 era.

The answer is, of course, no, you don't spin up an entire instance of Oracle to run a single unit test. You run against a local instance, and you use whatever tricks your-RDBMS-of-choice gives you to make resetting the test tables to a known-good state extremely fast. That way you can have your tests running continuously, giving you fast feedback as you develop, only suffering db overheads when you actually need to run tests that hit it.

If your chosen stack makes it difficult to do this, it's worth asking why.

Re: When to Mock

#74

Post by commandlinefan »

swat535 wrote: post #70
commandlinefan wrote: post #21

Earlier quoted context omitted.

On HN, it's not debatable: you should definitely do it. Everywhere I've ever worked, it's also not debatable: no time or effort should be "wasted" writing unit tests, that's what QA is for.

> no time or effort should be "wasted" writing unit tests, that's what QA is for. This is not my experience at all, everywhere I worked unit tests are required. I don't agree at all, writing unit tests is not a waste of time, this sentiment comes from people who don't know how to avoid brittle tests. It's impossible for QA to test every single path in your code. Unless you wanna cover all of those in your slow end to…

Oh, you're preaching to the choir here, pal. It's incredibly frustrating. I have reason to hope, though... when I first started programming in the 90's, I would fight with my co-workers about using version control (I insisted on it, they said it was a waste of time). Now, people who oppose version control are off selling life insurance or whatever happens to people who probably shouldn't be writing computer programs.

Re: When to Mock

#75

Post by lifeisstillgood »

mabbo wrote: post #3

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…

If I am honest, at this point I would rather not use a mock, but a stub / injection.

I know this gets horrbily pedantic but its easier to see in code

def difficult_function(dbconn, a,b,c): dbconn.execute(Select * from tbl)

I would not want to mock the dbase at this point. Please can we instead do this

def difficult_function(dbconn, a,b,c): resultset_as_dict = dbconn.execute(select * from tbl) insider_function(a,b,c,results_set_as_dict)

def insider_function(a,b,c,results_set_as_dict): This can now be tested without mocks quite easily.

I think if you are doing 'difficult' stuff with a exernal database you are de facto, writing integration tests.

In fact i would say anything involving a database MUST be treated as a integration test. If it takes ten minutes thats fine - its an integration test.

If you want fast and external connections, use sqllite as part of your testing CI suite. But dont moan.

and do not use Mocks.

Re: When to Mock

#76

Post by anarazel »

harrisonjackson wrote: post #44

For a long time, we ran our Django test suite against an in-memory SQLite DB. It was super fast, which encouraged more tests to be written and a CI/CD process that allowed everyone to confidently ship code often. Our production database is postgres. We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we cou…

It obviously makes sense that creating a new PG instance will be slower than creating a sqlite database. There also is some inherent speed difference for simple queries just by virtue of sqlite not needing to context switch to a separate process and to marshal the query/results across process boundaries.

But if the difference is more substantial than those factors would suggest, I'd be interested to see if we can do something about it from the PG side.

Re: When to Mock

#77

Post by bluejekyll »

sanderjd wrote: post #33
bluejekyll wrote: post #6

Earlier quoted context omitted.

Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less…

Do people usually mock the database at the integration / functional level? I haven't seen that... I interpreted this to mean the unit test level, which similarly to the thread starter, seems crazy to me.

If you think of the DB like a service, then yes, people sometimes mock it. IMO, the danger comes when you start trying to mock things and act like a DB, i.e. you create some in-memory store to act like a DB. That's dangerous because then you're potentially introducing issues that are completely unrelated to the DB operations, and therefor not testing anything of value.

But it can be far cheaper to generate data through a mocked interface, than say fill a DB with data, and test against that data-set. Obviously there are ways to structure your code such that the DB isn't part of the data flow at all, but sometimes existing code structure isn't perfect.

Re: When to Mock

#78

Post by bluejekyll »

pydry wrote: post #38
bluejekyll wrote: post #6

Earlier quoted context omitted.

Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less…

However, the further down the pyramid you go: * The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all). * The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the da…

With all the projects I've worked on, I've always found unit tests to be the best possible place to catch bugs and errors, because it's faster and easier to diagnose the root cause.

That said, they don't cover all test cases, which is why it's a pyramid. What I have found is that when an bug arises and is caught in an integration test, it's often beneficial to create a unit test that helps catch the same error before it you get to the integration test area, not always, but definitely if you have something that fails more than once in an area. Unit tests should never have false negatives, there's something wrong with the test if that is happening.

That being said, tests are designed around the code that is being tested. As technical debt and refactoring of existing code happens, you do often need to rework tests. Many people allow tests to go unrefactored, and they become their own set of technical debt, but that doesn't mean that they don't have value.

Re: When to Mock

#79

Post by Tomis02 »

allendoerfer wrote: post #41
Tomis02 wrote: post #25

Earlier quoted context omitted.

Unit tests are code, and all code has a cost. Writing unit tests without any benefit is harmful behaviour. Real world example: testing dumb getters/setters.

Writing getters and setters is harmf... - okay, I will stop before things escalate in here!

If you live in OOP land and have a natural aversion to directly accessing the data then by all means, write all the getters and setters you want. But testing that "int getX() { return X;}" actually returns X is pathological behaviour. It's harmful because 1) it costs to write 2) it costs to read 3) it takes up space, distracting from more important tests 4) it costs to run 5) prevents you from easily making changes (it's a brittle test). Can't really see any benefits; if you manage to screw up writing a dumb getter, how do you trust yourself that the test code isn't wrong as well?

Re: When to Mock

#80

Post by pydry »

bluejekyll wrote: post #78
pydry wrote: post #38

Earlier quoted context omitted.

However, the further down the pyramid you go: * The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all). * The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the da…

With all the projects I've worked on, I've always found unit tests to be the best possible place to catch bugs and errors, because it's faster and easier to diagnose the root cause. That said, they don't cover all test cases, which is why it's a pyramid. What I have found is that when an bug arises and is caught in an integration test, it's often beneficial to create a unit test that helps catch the same error before…

>With all the projects I've worked on, I've always found unit tests to be the best possible place to catch bugs and errors

Have you considered that that might be due to the nature of the projects you've worked upon rather than the nature of unit tests themselves?

>it's often beneficial to create a unit test that helps catch the same error before it you get to the integration test area, not always, but definitely if you have something that fails more than once in an area.

It depends upon what the bug was. Sometimes it's possible. Sometimes "replicating" it with a unit test is expensive and largely pointless since the unit test won't catch that class of bug in the future and will break as soon as you change the code (e.g. I've seen people try to create unit tests that mimic race conditions before and the results were horrendous to read, pointless, and didn't even catch race conditions).

>That being said, tests are designed around the code that is being tested. As technical debt and refactoring of existing code happens, you do often need to rework tests. Many people allow tests to go unrefactored, and they become their own set of technical debt

The higher level and the more behavioral the tests are, the less they have to be changed when the code is refactored and the more confidence that they give you that the code actually works afterwards.

The absolute worst situation to be in is with a bunch of unit tests that are tightly coupled to code that needs refactoring. Those unit tests' breakages signal nothing except that you've changed some code and they demand expensive repairs before breaking again in the future - again, because an API endpoint was refactored, not because a bug was introduced.

Post reply on HN