Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Integration tests are a symptom of poor design

facebook.com

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Integration tests are a symptom of poor design

#31

Post by pavement »

WorldMaker wrote: post #17
pavement wrote: post #8

Networked resources have listeners. Listeners have ears. Ears have wax. A unit test verifies that your Q-tip is made of cotton, and that the cotton is soft and small enough to fit inside an ear to a depth that will fetch wax. Another unit test might confirm that a swabbed substance is actually ear wax, by reporting the qualities of a verified sample of earwax, and maybe also a sample of candle wax as a true negative.…

That's an interesting choice of analogy given that boxes of Q-Tips warn you to never try cleaning your ears with a Q-Tip and so do most professionals: http://time.com/4290668/q-tip-ear-wax-removal/ Current professional wisdom is that you shouldn't even bother cleaning your ears because wax buildup is healthier and the body's own wax removal process is good enough.

Come on, people. Work with me, work with me.

Re: Integration tests are a symptom of poor design

#32

Post by pavement »

aeorgnoieang wrote: post #15
pavement wrote: post #8

Networked resources have listeners. Listeners have ears. Ears have wax. A unit test verifies that your Q-tip is made of cotton, and that the cotton is soft and small enough to fit inside an ear to a depth that will fetch wax. Another unit test might confirm that a swabbed substance is actually ear wax, by reporting the qualities of a verified sample of earwax, and maybe also a sample of candle wax as a true negative.…

This was a good metaphor but I don't understand the last line. Our tests can never 'really' clean 'real ears' can they? [Actually, that's kind of an interesting idea. Send a real order thru your inventory, order management, etc. systems ...]

Payment processors frequently conduct "no-capture" test transactions for authorizations of possibly $1, before tokenizing CC info. Arguably, this follows the pattern of an integration test.

Ecommerce providers often follow a process of cycling an immaterial test product through a real credit card for pennies, with a petty cash credit card, to exercise the purchase/refund cycle, end-to-end, depending on the activities of the project. A runaway batch process with an infinite loop of purchases is bad, infinite refunds is even worse.

Sometimes these tests are needed to ensure that the service account exists on the third party system, is recognized, and has limited permissions for a fixed, restricted set of API calls. The tests might be run only during a release deployment, against live production, but more frequently against a third party sandbox host while coding. This is a very common pattern.

Re: Integration tests are a symptom of poor design

#33

Post by Chaebixi »

aeorgnoieang wrote: post #14
ardit33 wrote: post #9

I can't agree with this. In his example (Transactions), he assumes that there is perfect knowledge of the underlying data and perfect knowledge on the assumptions made by the teams in the corner cases of each module, which is not the case in practice. Very very often, (one of the most common source of errors in programming large systems), two parts of a program are doing the "right" thing, within their assumption, bu…

I don't think you're really disagreeing with Kent. And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain. And they can be more brittle than your actual code. If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? Maybe! But the more 'real' the test i…

> And they can be more brittle than your actual code.

The brittlest tests I've ever seen have been mock-heavy unit tests. Some of them amount to little more than file-change detectors. Those tests can fail massively even when everything still actually works. They cry wolf so often the team ends up trained to unthinkingly change them so they "go green."

> But I think I get what he's getting at in terms of – ideally – wanting to minimize those kinds of tests by transforming previously unreliable parts of the system into something we can use "as a given" (like integer arithmetic).

A noble goal, but practically not a lot different from saying we should aim for formally verified software. After all, it's what we have for computer arithmetic example: http://www.cl.cam.ac.uk/~jrh13/slides/jnao-02jun06/slides.pd... (2006):

> Intel uses formal verification quite extensively, e.g. ... verification of...floating-point unit[s]...

Re: Integration tests are a symptom of poor design

#34

Post by crdoconnor »

aeorgnoieang wrote: post #14
ardit33 wrote: post #9

I can't agree with this. In his example (Transactions), he assumes that there is perfect knowledge of the underlying data and perfect knowledge on the assumptions made by the teams in the corner cases of each module, which is not the case in practice. Very very often, (one of the most common source of errors in programming large systems), two parts of a program are doing the "right" thing, within their assumption, bu…

I don't think you're really disagreeing with Kent. And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain. And they can be more brittle than your actual code. If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? Maybe! But the more 'real' the test i…

>But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies.

A strangely underappreciated facet of being a more 'real test' is that you catch more 'real' bugs.

>should your integration test actually make a (test) request to the real live service endpoints?

Ideally you should be able to run your integration test in a mode that mocks it and a mode that calls the real live service so that you can see problems in your code and (and avoid brittleness) and problems with the service, and distinguish them.

Re: Integration tests are a symptom of poor design

#35

Post by jondubois »

jondubois wrote: post #30

Different people tend to have completely different ideas about what an integration test is. At one extreme, some people think it means that you have your entire system running and then you have some headless bot simulate clicks and evaluate the system's behavior based on the UI... From my point of view, any test that traverses through the logic of more than one kind of object is an integration test. Primitive types d…

...sign. Isn't that the whole idea behind keeping methods short and classes high cohesion?

The simpler your unit of code is, the less useful your unit test becomes. By that definition, I reach the opposite conclusion as the author's.

The problem with unit tests is that they assume that most of the complexity lies in individual functions.

From my experience, most of the complexity and unpredictability of software lies in the wiring logic between components.

Most difficult software bugs are not the result of a function not working correctly - Most of the time, it is simply the result of a function not being used correctly... E.g. Someone make an incorrect assumpti

Re: Integration tests are a symptom of poor design

#36

Post by moomin »

KentBeck wrote: post #5
moomin wrote: post #4

I think the problem I have with this is the idea that the design is static. In practice, if you’ve got five layers, their exact behaviour can be subject to subtle change. Integration tests (not necessarily whole system tests) are invaluable for finding where changed assumptions actually break required behaviour. This, in turn, can inform a better design. Never thought I’d be disagreeing with Kent Beck about anything…

How is this a disagreement?

If you don’t categorise it as one I guess we’re good. :)

Re: Integration tests are a symptom of poor design

#37

Post by Chaebixi »

aeorgnoieang wrote: post #14
ardit33 wrote: post #9

I can't agree with this. In his example (Transactions), he assumes that there is perfect knowledge of the underlying data and perfect knowledge on the assumptions made by the teams in the corner cases of each module, which is not the case in practice. Very very often, (one of the most common source of errors in programming large systems), two parts of a program are doing the "right" thing, within their assumption, bu…

I don't think you're really disagreeing with Kent. And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain. And they can be more brittle than your actual code. If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? Maybe! But the more 'real' the test i…

> But the more 'real' the test is, the less of your own test it covers relative to all of the other external dependencies.

But the more 'real' the test is, the more of your own assumptions get about how your dependencies work get tested. Assumptions are where most bugs are born and hide.

PS: Sorry for the multiple replies. HN is being glitchy and my previous comment kept getting truncated.

Re: Integration tests are a symptom of poor design

#38

Post by WalterSear »

aeorgnoieang wrote: post #14
ardit33 wrote: post #9

I can't agree with this. In his example (Transactions), he assumes that there is perfect knowledge of the underlying data and perfect knowledge on the assumptions made by the teams in the corner cases of each module, which is not the case in practice. Very very often, (one of the most common source of errors in programming large systems), two parts of a program are doing the "right" thing, within their assumption, bu…

I don't think you're really disagreeing with Kent. And I haven't found that integration (system) tests have a higher ROI – because they're usually much more expensive to write and maintain. And they can be more brittle than your actual code. If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints? Maybe! But the more 'real' the test i…

> If your code depends on an external service, should your integration test actually make a (test) request to the real live service endpoints?

That's a terrible idea. You've now coupled your tests to someone else's code.

Test the endpoints separately. Wrap the endpoints to allow you convenient dependency injection. Test the wrapper. This wrapper is ideally a very thin layer of boilerplate, that you won't ever touch very much, and shouldn't take much time to write or test.

But this is the key, however: use the data specifying the expected outputs of your third party API as input parameters to the test of the wrapper, and expected outputs from the wrapper as parameters for your integration tests. Keep each set of data in one place, so that changing it for one test, changes it for the others.

By using shared test datasets in this way, you prevent modules from getting out of synch with each other, and can still replace/update parts in separation.

Re: Integration tests are a symptom of poor design

#39

Post by crispinb »

mabbo wrote: post #29

The presumption here is that you are a team of Kent Beck's. Mine isn't. Yes, with a lot of work and time, we can come up with very good abstractions so that all code is easy to understand and follow and unit tests are all we need. But we don't have that time, nor often the skill needed. What we need is a running product, today, for our customers. Integration tests are a crutch. Some of us need crutches. And I don't f…

> The presumption here is that you are a team of Kent Beck's

Not really. He explicitly says 'I’m happy to write integration tests. I insist on it'; in other words Kent Beck is admitting he doesn't always know how to perfect the underlying layers' design so as to push 'integration tests' towards being unit tests. But if he's right about the nature of the distinction, ie. that apparent 'integration tests' are really a symptom of underlying design limitations, it's surely a good thing to be aware of?

There are many areas where I do know my own code is problematic, but for lack of time or competence I can't improve it now. I don't believe I would be better off not knowing there was a problem.

Re: Integration tests are a symptom of poor design

#40

Post by throwaway_f8ed »

Disclaimer: I mean no offense.

I am yet to see one production-ready real-world code example from such gurus of design of the past. Kent Beck, Bjarne Stroustrup, Scott Meyers.

I think that in 2017 you shouldn't be entitled for your opinion about design if you have nothing to back up your cases. Sadly, this also covers me with this throw-away account, but bear with me for a while.

When they started their journey the world was different and chains of thought that led them to their current points of view are long obsolete.

We are living in a different world with different values. My own experience with integration tests led me to believe that I cannot afford to skip on them. There are far more moving parts nowadays and it's better to know that something introduced breaking changes from tests than from angry customers. Something along long chain of dependencies that you did not even consciously knew you are depending on.

Now, some guy who is known only for some books he wrote in times of yore comes in and tells me I am doing it wrong, only on basis of his non-existent experience and unfounded authority? We already know what comes out of unsoliticed advice like that. We wasted decades on OOP modeling, UML and tons of other things that never came off.

There are now widespread ideas that he either conceived or popularized, but even these mutated to the point of unrecognizability.

How about we will move on?

Post reply on HN