Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Integration tests are a symptom of poor design

facebook.com

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Integration tests are a symptom of poor design

#101

Post by spronkey »

KentBeck wrote: post #95
mabbo wrote: post #29

The presumption here is that you are a team of Kent Beck's. Mine isn't. Yes, with a lot of work and time, we can come up with very good abstractions so that all code is easy to understand and follow and unit tests are all we need. But we don't have that time, nor often the skill needed. What we need is a running product, today, for our customers. Integration tests are a crutch. Some of us need crutches. And I don't f…

Hang on a minute. I'm both Superprogrammer and a washed up old has-been? This is getting confusing. There's no shame for me in using integration tests. They just hint at an alternate universe where the design is different and they either disappear entirely or become unit tests. So today isn't the day that happens. Okay. "Perfect" is a verb.

I'm struggling to understand how the transition from integration test to unit test might manifest in terms of a real world refactoring.

My 'integration tests' almost always cross the boundaries of two or more (theoretically) well-defined APIs. If they're within those boundaries, well, they're not really integrating anything, so I would just consider them a unit test.

Do you have any examples of this?

Re: Integration tests are a symptom of poor design

#102

Post by WalterSear »

skybrian wrote: post #61
WalterSear wrote: post #48

Earlier quoted context omitted.

>It's a terrible idea not to. If your code isn't coupled to someone else's code your code isn't doing anything useful. Paradoxically, you want your tests separate from the external inputs, because complete control over the inputs is required to systematically and programmatically hit every edge case with every pass of your test suite. You also need the modularity in your tests and code, so you can fix and replace 3rd…

You're missing a fundamental difference in philosophy. Why should we only care about bugs in our code? They aren't the only bugs our users care about. A browser bug (for example) may not be our fault, but it is our problem. Therefore, we need to run our tests in real browsers to make sure it actually works, and keep running those tests on new browser releases to find any new bugs. Similarly, mobile apps need to be te…

I beleive our disagreement is fundamentally one of semantics. However, I'll concur that I have not used this method for front end or mobile or embedded code yet, and there may be problems that I haven't anticipated.

That said, I think you miss my point - I'm not saying don't test in a real-world environment: I'm saying decouple your program logic from its inputs, and enforce shared test data between top-level edge and the integration test suites to ensure end-to-end integrity (and do the same for every other part of your code that interacts with another part of your code, fwiw).

> Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper.

So, test in the browser, or on the app. Just test the endpoints separate from your integration test, and use the explicitly stated output of integration test as the input parameters for your browser tests. I'd go far as to argue that browsers and hardware - where you have the greatest concerns about the stability of your 3rd party APIs, are actually the circumstances where separating edge testing from integration makes the most sense.

Perhaps a better way to explain my process is to think of my set of edge and integration tests as the equivalent of your 'integration' test suite, with the addition of specific contracts (the shared data) to ensure that interaction between modules is also controlled for. You are still testing end-to-end and you are still testing with real-world data, since that's what the edge tests will be returning. You are just explicitly stating what you expect at each boundary of the code.

Re: Integration tests are a symptom of poor design

#103

Post by WalterSear »

mcphage wrote: post #79
WalterSear wrote: post #55

Earlier quoted context omitted.

> You've now coupled your tests to someone else's code.

Right, but what I'm saying is, it's even worse than that. Not only are your tests coupled to someone else's code, but your code itself is so coupled.

The trivial realities of the actual world may chill my heart to the bone, but even I can accept this sad inevitability ;)

The thing is, the fact that your code relies on someone else's, and that their code can change out from under you is exactly why you need separation in your tests and modularity in your code. Using the system I am describing, you are still working with 'real world' data - you are just asserting it in advance (since the real world inputs correlated with the inputs of your integration suite via the edge tests) and ring fencing your endpoints.

I've actually been part of a software team driven wild by unknown changes to infrastructure endpoints, causing failure of the integration test suite. And then by the brittleness of an integration suite too tightly coupled to inputs.

I've also worked with test suites that failed when run after business hours, but only during Atlantic DST, for the same reasons. :|

Re: Integration tests are a symptom of poor design

#104

Post by crdoconnor »

bonesss wrote: post #100
crdoconnor wrote: post #76

Earlier quoted context omitted.

What you are arguing for is that code should be loosely coupled. I generally agree with this (although, like DRY, its validity is context sensitive and one can take this approach too far). This is an orthogonal question to whether code should be tested realistically or not, however. You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible to iso…

> ...You want to test your code against real external inputs for additional realism as well as mocks that are as realistic as possible... Ding ding: it's not either/or, it's both/and across a spectrum from dumb to smart to dumb again. If you use a google rest API in something serious you want to be able to model the error conditions of that API, you also need to verify the current status and content of that API, you…

>Ding ding: it's not either/or, it's both

Yes, that was the point I was making.

>Loose coupling and DI are important to strike the right balance

DI is useful sometimes (e.g. when you have a set of modules you want to hotswap with one another), but it's often just an overused crutch to deal with the inability of unit tests to couple to real things - like an actual REST API endpoints over a loopback interface.

The whole idea that unit tests drive "good code" by making you DI all the things is a pile of shit. Unit tests just make it painful to not do DI because unit tests are themselves a form of tight coupling.

Re: Integration tests are a symptom of poor design

#105

Post by crdoconnor »

WalterSear wrote: post #102
skybrian wrote: post #61

Earlier quoted context omitted.

You're missing a fundamental difference in philosophy. Why should we only care about bugs in our code? They aren't the only bugs our users care about. A browser bug (for example) may not be our fault, but it is our problem. Therefore, we need to run our tests in real browsers to make sure it actually works, and keep running those tests on new browser releases to find any new bugs. Similarly, mobile apps need to be te…

I beleive our disagreement is fundamentally one of semantics. However, I'll concur that I have not used this method for front end or mobile or embedded code yet, and there may be problems that I haven't anticipated. That said, I think you miss my point - I'm not saying don't test in a real-world environment: I'm saying decouple your program logic from its inputs, and enforce shared test data between top-level edge an…

>So, test in the browser, or on the app. Just test the endpoints separate from your integration test

Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code.

That means two sets of tests (add person browser test / add person API test) that will nearly always break on the same bugs. That means two sets of tests to maintain when you change the code.

It's not that you shouldn't sometimes drill down to test code at a lower level, it's that when you do, you should be drilling down to a reusable abstraction that is independent of the higher level code.

Re: Integration tests are a symptom of poor design

#106

Post by WalterSear »

crdoconnor wrote: post #105
WalterSear wrote: post #102

Earlier quoted context omitted.

I beleive our disagreement is fundamentally one of semantics. However, I'll concur that I have not used this method for front end or mobile or embedded code yet, and there may be problems that I haven't anticipated. That said, I think you miss my point - I'm not saying don't test in a real-world environment: I'm saying decouple your program logic from its inputs, and enforce shared test data between top-level edge an…

>So, test in the browser, or on the app. Just test the endpoints separate from your integration test Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code. That means two sets of tests (add person browser test / add person API test) that will nearly always break on the same bugs. That means two sets of tests to maintain when you chang…

> Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code.

Except the endpoints are a wrapper. They aren't doing anything but acting as a pass through for injection. So, you are talking about a few minutes of boilerplate that will only change when the API changes, as a small price to pay for the ability to inject your endpoint dependencies (or their stubs) without the mess and complexity of proxying.

If it's so objectionable, handle injection some other way, without the wrapper: it's not fundamental to the idea of using data contracts to ensure correct communication between modules, while keeping the edge and core integration suites modularized.

Re: Integration tests are a symptom of poor design

#107

Post by crdoconnor »

WalterSear wrote: post #106
crdoconnor wrote: post #105

Earlier quoted context omitted.

>So, test in the browser, or on the app. Just test the endpoints separate from your integration test Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code. That means two sets of tests (add person browser test / add person API test) that will nearly always break on the same bugs. That means two sets of tests to maintain when you chang…

> Unless you're testing different behavior when you're testing the endpoints, what you're doing is essentially writing duplicated test code. Except the endpoints are a wrapper. They aren't doing anything but acting as a pass through for injection. So, you are talking about a few minutes of boilerplate that will only change when the API changes, as a small price to pay for the ability to inject your endpoint dependenc…

>without the mess and complexity of proxying

Proxying is less of a mess and less complex than DI. It requires zero changes to your code and can be done with off the shelf tools, it will continue to work if you rewrite your endpoint in a different language and you easily can do stuff like mimic high load or flaky networks to see what happens to your system.

>If it's so objectionable, handle injection some other way

Why do DI at all if you're just doing it for testing purposes?

Re: Integration tests are a symptom of poor design

#108

Post by mcphage »

WalterSear wrote: post #103
mcphage wrote: post #79

Earlier quoted context omitted.

Right, but what I'm saying is, it's even worse than that. Not only are your tests coupled to someone else's code, but your code itself is so coupled.

The trivial realities of the actual world may chill my heart to the bone, but even I can accept this sad inevitability ;) The thing is, the fact that your code relies on someone else's, and that their code can change out from under you is exactly why you need separation in your tests and modularity in your code. Using the system I am describing, you are still working with 'real world' data - you are just asserting it…

> I've actually been part of a software team driven wild by unknown changes to infrastructure endpoints, causing failure of the integration test suite.

Now imagine that your tests hadn't been so tightly coupled, and there were unknown changes to infrastructure endpoints. Suddenly your code starts failing, and you can't tell why, because your test suite is all green.

Post reply on HN