Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When to Mock

enterprisecraftsmanship.com

Post reply on HN
Print view View original

51–60 of 82 posts

Re: When to Mock

#51

Post by amw-zero »

grey-area wrote: post #7
mabbo wrote: post #3

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…

When you're developing, aren't you running a db server with a dev database for that? Just add another separate test db and use that for tests where appropriate. Also, most unit tests don't need a db, it's more useful for integration tests.

Hitting a database in tests is an order of magnitude slower and also involves worrying about a lot more code, i.e. the database engine. You may or may not want to take on that burden.

Re: When to Mock

#52

Post by taffer »

harrisonjackson wrote: post #44

For a long time, we ran our Django test suite against an in-memory SQLite DB. It was super fast, which encouraged more tests to be written and a CI/CD process that allowed everyone to confidently ship code often. Our production database is postgres. We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we cou…

Can't you use transactions to set up and tear down fixtures? That should be a lot faster than spinning up a new database every time.

Re: When to Mock

#53

Post by BenoitP »

mabbo wrote: post #3

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…

> should spin up an entire instance of Oracle

I'm sort of doing that at the moment, with postgres and series of tests. But it is still useful for unit testing too: pull up the right CREATE tables, INSERT your test data, execute your tests, then drop tables (lots of safeguards here), and repeat. The container loads up in 1.5 sec, and all my tests (~100) are done in 10 seconds.

It's been great in my use case. I'm on postgres, Java, and using testcontainers. They have handy containers available, not limited to RDBMS and not even limited to databases; and here is Oracle Express edition, which should be enough for most tests:

https://www.testcontainers.org/modules/databases/oraclexe/

Re: When to Mock

#54

Post by stickfigure »

mabbo wrote: post #3

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…

IMO, the classic test pyramid with integration tests at the narrow top and unit tests at the wide bottom is inverted. We test software that way because integration tests are hard and unit tests are easy, but integration tests provide more value than unit tests. It's the streetlight effect: "Did you lose your car keys here?" "No, but the light is much better here."

I built some framework code so that integration tests as easy to write as unit tests. Almost all of my tests are "integration tests". I never mock the database. My test harness clones a template database at the start of each test; that database is maintained with migrations just like every other database. I run against test accounts at all the 3rd party services I use. It's not super fast - a full CI run takes about 15 minutes. And occasionally a 3rd party service will cause a test to flake. But it's fast enough and most importantly it's thorough.

I still have some unit tests that run without the expensive setup harness, but they're for components that have a lot of algorithmic complexity. Anything that touches the database gets a real instance.

This works pretty well with Postgres, which is free and I can run locally and has a fast db clone operation. YMMV with other databases.

Re: When to Mock

#55

Post by taffer »

malinens wrote: post #50
harrisonjackson wrote: post #44

For a long time, we ran our Django test suite against an in-memory SQLite DB. It was super fast, which encouraged more tests to be written and a CI/CD process that allowed everyone to confidently ship code often. Our production database is postgres. We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we cou…

if you keep using sqlite for tests it forces Your database logic to be universal. You could confidently switch to other db like mysql any time.

This also means that you have to stick to the lowest common denominator between all databases.

Re: When to Mock

#56

Post by amw-zero »

pydry wrote: post #38
bluejekyll wrote: post #6

Earlier quoted context omitted.

Right, the testing pyramid. Your largest set of tests should be a foundation of unit tests that run quickly. Then integration and or functional tests, which is the section this is talking about. They are still necessary, then move up the stack and you get into the end-to-end tests of a fully running system. The reason isn’t only speed but also signal to noise ratio. The further up the testing pyramid you go, the less…

However, the further down the pyramid you go: * The more unrealistic the tests generally become (larger % of false positives - tests that fail when they shouldn't - and false negatives - unit tests that simply don't catch bugs at all). * The less reusable the test infrastructure becomes. Stubbing/mocking individual method calls to the database is an ongoing cost of development whereas building scripts to start the da…

I've started to agree with this view of the test pyramid as well. This is a great video overall, but here's a part where Aslak Hellesoy (creator of Cucumber) is talking about how a better way to think about the test pyramid is as a [spectrum of speed & reliability](https://www.youtube.com/watch?v=PE_1nh0DdbY&t=12m55s)

If a test isn't a pure unit test, where all collaborators are stubbed out, but it is still fast and reliable, it is still a very valuable test. Possibly preferable since testing multiple collaborating objects / functions provides more confidence than just testing one by itself.

Re: When to Mock

#57

Post by squiggleblaz »

jaitsu wrote: post #5
mabbo wrote: post #3

What bothers me about this article is that it doesn't really present alternatives or real-world examples of how to do what the author is saying. > Use real instances of managed dependencies in tests. Is the author suggesting I should spin up an entire instance of Oracle to run a single 50ms unit test? Because I often want to run my unit tests very frequently as I develop, ensuring I'm still green, etc. And my code ma…

Writing unit tests is debatable?

On legacy code? Alas. The amount of changes you might have to make to make the legacy code unit testable versus the benefit of unit tests can be exceedingly significant.

For instance, if the implementation involves calling up a bunch of value objects from a database, each of which do the same, and all of the code is inhouse so there is no standard mock or stub libraries available, adding unit tests is tantamount to rewriting the whole system without tests.

Existing codebases can also be too complicated for a few people to formalise into unit tests. The algorithm itself might be simple, but again, to discover that, you need to rewrite the whole system without tests. (You can add tests, of course, but when you're adding tests, you're making an assertion you cannot prove. Since you don't know what the code is meant to do, you don't know whether the tests are complete or even accurate.)

Once you're coding in a world where tests passing or tests failing has almost no predictive value on success or failure in release, you're working in a self-fulfilling prophecy where the code will never be tested because the tests literally make things worse.

Many codebases clearly do not have any automated tests at all, but unit tests can be the hardest to add onto a system after the fact.

Re: When to Mock

#59

Post by squiggleblaz »

bob1029 wrote: post #45

Our trick is to use SQLite to completely side-step the concern of maintaining any sort of central testing database. Each developer can clone a fresh copy, load the .sln, hit F5, and every database required by the application is immediately available within the same process. There is absolutely no other software required to be installed. This also makes it infinitely easier to coordinate complex schema changes. Each d…

Are you using SQLite as your main production datastore or you're using sqlite as a standin for something else? It sounds like production. One db file or several? How big? Can you summarise the non-test benefits and costs?

Re: When to Mock

#60

Post by stickfigure »

harrisonjackson wrote: post #44

For a long time, we ran our Django test suite against an in-memory SQLite DB. It was super fast, which encouraged more tests to be written and a CI/CD process that allowed everyone to confidently ship code often. Our production database is postgres. We kept bumping against things we wanted to do in the application code that worked well with postgres, but would fail in sqlite. We limited our development so that we cou…

It isn't clear if you're already doing this, but Postgres has a 'create database from template' feature. You can initialize your template database with migrations once for a whole test suite, then clone a new database at the start of each test.

It's quite fast. I run almost all of my tests this way.

I agree, forcing your app into the lowest common denominator of portable SQL is crippling. JSONB columns in particular are extremely useful in Postgres.

Post reply on HN