Stop mocking your system • Hacker News Board

Re: Stop mocking your system

#161

Post by JoshuaDavid » Mon, Jul 05, 2021, 4:05 AM UTC

bluGill wrote: post #132

Aeolun wrote: post #129

Earlier quoted context omitted.

Lolwut? Unit tests are one of the few things that give me confidence in my refactored code. Integration tests are nice, but they don’t give me enough confidence about the specific part of the system I’m altering.

Until you realize that your api is wrong. So long as you got your api right up front unit tests make it easy to refactor. As soon as the api needs nontrivial changes you are sunk.

If your api is wrong, and you have unit tests for that wrong api, and you have time to rewrite it the right way, you can write a set of methods with the correct api which is a shim for the wrong api (you don't care about perf here), write your refactored version, and switch your now-tested shim to the correct api (and then probably keep it as a reference for the new way of doing things).

If you find yourself doing this frequently though unit tests might not be ideal for your situation.

Re: Stop mocking your system

#162

Post by catwind7 » Mon, Jul 05, 2021, 4:09 AM UTC

tuxie_ wrote: post #72

catwind7 wrote: post #3

Earlier quoted context omitted.

what does unit testing have to do with whether or not you instrument the test with fake responses? those points of contact that you're mocking out at the perimeter, that data will sometimes need to reach a particular function through a collaborator...which you may want to mock? sometimes the dependency is not a third party, but it may be code that requires a ton of setup (as mentioned in article) that's not worth the…

I'm sorry, I did not intend to offend anyone obviously. Needless to say, this is just my opinion condensed in a sentence (therefore lacking a lot of context, which I should have provided). I was not aiming to define what a unit test is, more like when it stops being a unit test (which I thought it would be an easier agreement to reach than a definition of what is, but I guess I underestimated the task). My point was…

this convo has made me realize that our terms for testing do not pair well with actual testing practices, which i find easier to conceptualize in terms of "coarseness" relative to fundamental units of work in a system as opposed to this binary notion of unit versus integration.

i might write a blog post about this!

Re: Stop mocking your system

#163

Post by alecbz » Mon, Jul 05, 2021, 4:10 AM UTC

rualca wrote: post #96

alecbz wrote: post #51

True "unit" tests are: * faster to run * give you less confidence in the correctness of the system (per time spent writing them) * when they fail, give you more information about where the failure is The more integration-y/e2e-y a test is, the more it strays from this: slower to run, more confidence that the system is correct, less info about where the failures are. I think people have learned to undervalue the prope…

> I think people have learned to undervalue the properties of integration tests and overvalue the properties of unit tests. Is it nice to know exactly what's broken based on the test failure? Sure. Is it _as_ nice as having confidence that the whole system is working? Probably not, in a lot of cases. This comment alone demonstrates a collosal misunderstanding regarding unit tests. Unit tests are not used to verify if…

Are you saying unit tests are not meant (in part) to detect bugs?

Re: Stop mocking your system

#164

Post by EdSharkey » Mon, Jul 05, 2021, 4:28 AM UTC

Chris_Newton wrote: post #69

EdSharkey wrote: post #40

Earlier quoted context omitted.

I think so too. Unfortunately, integration tests are too slow, so the practice doesn't scale if one is trying to TDD. Insinuating integration testing into every user story will lead to friction. Test run times will balloon, cycle times will get extended and resentment will grow for the test suite and the team's testing regime. If your test suites cannot complete quickly (seconds), then they cost more than they're wor…

Unfortunately, integration tests are too slow, so the practice doesn't scale if one is trying to TDD. If integration tests get more useful outcomes than unit tests in some situation but TDD only works well with unit tests, maybe that means TDD isn’t the best process to use in that situation. Isn’t the essence of agile development to recognise what is working well and what is not, so you can make changes where they ar…

Integration tests, for my team at least, really are the workhorses for determining if we have shipping software or have more work to do. So much of our deployed code relies on configuration and packaging to function properly that units just don't test!

There have been so many errors in our coding over the years that only came out in our integration tests. I agree with you that, given my team's current maturity level, going slower and having multi-minute test runs is okay given our incredibly low defect rate in PROD.

Those long running test suites have created too much friction for new feature development. So, while the way we are working is comfy and life is easy, I feel we can do much better.

I believe it is possible to have rapid test cycles with our outside-in style TDD by keeping the added safety of integration tests but run the majority of them on the CI/CD server only, and change the way we unit test to rely less on mocks which should get us the 2-5 second inner loop cycle times we wish to hit.

Re: Stop mocking your system

#165

Post by EdSharkey » Mon, Jul 05, 2021, 5:09 AM UTC

pydry wrote: post #43

EdSharkey wrote: post #40

Earlier quoted context omitted.

I think so too. Unfortunately, integration tests are too slow, so the practice doesn't scale if one is trying to TDD. Insinuating integration testing into every user story will lead to friction. Test run times will balloon, cycle times will get extended and resentment will grow for the test suite and the team's testing regime. If your test suites cannot complete quickly (seconds), then they cost more than they're wor…

I find I can do ITDD effectively with integration test times of What makes it work is pairing it with a REPL. That way I can have an "outer" loop that triggers the REPL and then an "inner" loop where I can experiment in the area where I'm writing the code and get feedback quickly. I might run the outer loop just a few times: * At the beginning of the ticket * When I messed up the state with the REPL and when I want a…

I don't have easy REPL opportunities for speeding up my cycles in the language/environment I have to work with. :/

Martin Fowler discussed it recently, note the section that discusses about sociable vs solitary tests: https://martinfowler.com/articles/2021-test-shapes.html

My team is oriented towards solitary. For unit tests, mocks stand-in for collaborators with the system under test. For integration tests, databases and queues are realistically simulated with locally running instances and downstream services we call are wiremocked with local configs pointing service clients to those mocks. It's an incredibly effective way to produce reliable, fully tested code. It's also incredibly costly execution time-wise.

James Shore has a nice take on sociable tests; I interpret what he's produced is like a mishmash of unit and integration tests. I have trouble making my brain work that way, but I am intrigued given his impressively low test run times. There is something special there, I feel it just needs a sexy paint job for people to take notice. https://www.youtube.com/watch?v=jwbKSiqG0DI

I liked this blog post on the topic as well: https://merge-conflict.com/post/sociable-unit-testing/

Re: Stop mocking your system

#166

Post by EdSharkey » Mon, Jul 05, 2021, 5:18 AM UTC

AlphaSite wrote: post #79

EdSharkey wrote: post #40

Earlier quoted context omitted.

I think so too. Unfortunately, integration tests are too slow, so the practice doesn't scale if one is trying to TDD. Insinuating integration testing into every user story will lead to friction. Test run times will balloon, cycle times will get extended and resentment will grow for the test suite and the team's testing regime. If your test suites cannot complete quickly (seconds), then they cost more than they're wor…

Why run your whole test suite every time? Chances are you’re only really interested in a small set of them, just let those run continuously.

Definitely. Most of our developers do do that. We typically only run a subset corresponding to just the service containing the feature we're focused on.

On our larger services, unit tests take 20-30 seconds to run (WAY too slow) and integration tests take 2-4 minutes to run. In both suites we have hundreds of tests. There's no excuse for how slow the unit tests are. I chalk it up to the heavy reflection in the mocking frameworks we rely on and inefficient use of those mocks. Got to either optimize or decrease the use of mocks in our tests.

Re: Stop mocking your system

#167

Post by rad_gruchalski » Mon, Jul 05, 2021, 5:41 AM UTC

oriolid wrote: post #121

rad_gruchalski wrote: post #16

I agree with the premise of this post. Instead of mocking, design your components in such a way that they are easy to spin up in tests. Or use containerized versions of services. I started disliking the idea of mocks a few years ago. I was writing a system based on the Play framework. Play framework used to (still does?) come with a dedicated integration testing environment. The problem with it was that the setup of…

The problem is not spinning up services, but setting them up to fail in the way your test case needs them to fail. Testing the happy path isn't that useful, just trying out the system should show that they work. Edge cases and error conditions are what needs testing.

There is no problem with that. A program execution is a result of a given input. Error condition is therefore a result of a set of conditions leading to a fault. There faults can be recreated. The only two things I was not able to test reliably so far are: network latency / failure and disk latency. That is possible with firecracker but I haven’t had a need to go so elaborate yet.

Re: Stop mocking your system

#168

Post by sdeframond » Mon, Jul 05, 2021, 6:13 AM UTC

teddyh wrote: post #85

sdeframond wrote: post #65

Recently, I thought about a process I call "Test Coverage Driven Testing". It is similar to TDD, but more adapted to when we write tests after the code (you know you do too, at least occasionally, don't lie). It goes roughly like this: - write one integration test for the "happy path". - using some test coverage report, identify untested cases. - write unit test for those. I find I helps me find a good balance betwee…

> - using some test coverage report, identify untested cases. From what I understand, that is not reliable; a line of code can be “covered” – i.e. executed – but still not be tested under all circumstances. If you have pre-existing code you need to write tests for, what you need is probably a tool for mutation testing.

This is right. This is not a reliable approach since merely calling a function does not mean I tested all of its edge cases. And if my code depends on a 3rd party lib, I might not even have access to this lib's source.

OTOH, aiming for 100% reliability and coverage is waay too expensive for most business app. This is not like embedded software for a plane, where lives are at risk. I usually aim for a 80-90% coverage of my own code, plus a regression test for each bug actually reported.

And by the way, if you really want zero error (planes, trains, cars etc), TDD is not enough anyway.

Re: Stop mocking your system

#169

Post by cemoktra » Mon, Jul 05, 2021, 6:30 AM UTC

I oppose to that. We have unit tests calling API endpoints of third party sandbox. Once a week all of our CI fails cause third party sandbox is down

Re: Stop mocking your system

#170

Post by flukus » Mon, Jul 05, 2021, 7:17 AM UTC

catwind7 wrote: post #162

tuxie_ wrote: post #72

Earlier quoted context omitted.

I'm sorry, I did not intend to offend anyone obviously. Needless to say, this is just my opinion condensed in a sentence (therefore lacking a lot of context, which I should have provided). I was not aiming to define what a unit test is, more like when it stops being a unit test (which I thought it would be an easier agreement to reach than a definition of what is, but I guess I underestimated the task). My point was…

this convo has made me realize that our terms for testing do not pair well with actual testing practices, which i find easier to conceptualize in terms of "coarseness" relative to fundamental units of work in a system as opposed to this binary notion of unit versus integration. i might write a blog post about this!

More and more I'm seeing "unit testing" becoming the generic term for what we used to just call plain old testing. A lot of companies had never had anything remotely close to unit testing but called all their automated tests unit tests, at least in part out of ignorance of the difference. My current company has gone a step further and calls their extremely manual and poorly defined QA test plans unit testing.

As for my 2 cents, I find single assert principle helpful as it helps narrow down the unit of behavior your actually testing. I don't care if a test covers a single function or half the code base, as long as it's clear why it is specifically testing for and what has gone wrong if it fails.