Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The testing pyramid should look more like a crab

changelog.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: The testing pyramid should look more like a crab

#41

Post by Tainnor »

musingsole wrote: post #15
raziel2p wrote: post #4

> Well, the unit tests can find you a few logical errors, which is great. I write unit tests for that all the time. But all possible sources of error are discovered by end-to-end tests. Citation needed, sounds anecdotal. If this really is how your app behaves then you're going to run into problems eventually because end-to-end tests are slow as well as difficult to write+maintain, and you'll need hundreds (if not tho…

> end-to-end tests are slow as well as difficult to write+maintain Citation needed, sounds anecdotal. End-to-end tests should run as fast as any given use case (for most software, that means relatively quick), plus some set-up time as testing overhead. But that testing overhead is often self-inflicted and can be managed. I have anecdotes to prove it. For the speed to write end-to-end tests, shouldn't they be the easi…

I'm pretty sure more people care about the VAT being calculated correctly than about whether it's displayed on the left or on the right.

You're taking a reasonable point of view - code matters only inasmuch as it has an effect on users - and take it to an absurd extreme.

Re: The testing pyramid should look more like a crab

#42

Post by contravariant »

pydry wrote: post #18
raziel2p wrote: post #4

> Well, the unit tests can find you a few logical errors, which is great. I write unit tests for that all the time. But all possible sources of error are discovered by end-to-end tests. Citation needed, sounds anecdotal. If this really is how your app behaves then you're going to run into problems eventually because end-to-end tests are slow as well as difficult to write+maintain, and you'll need hundreds (if not tho…

>If your code is well-structure You've alluded to the real benefit of end to end tests here, IMHO. They make very few demands about how your code is structured. This is fantastic if you've just been handed a big ball of mud. There is a definite trade off between faster running tests and tests which don't need to be rewritten every time you change some code.

Is that a benefit of end-to-end tests? Seems more like encouraging more unit tests would discourage bad coding habits if anything.

Re: The testing pyramid should look more like a crab

#43

Post by closeparen »

pydry wrote: post #38
CivBase wrote: post #36

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a ma…

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

I think of it as double entry bookkeeping. You have to say what you mean two different ways, and those two phrasings have to be reconcilable. This is still a useful guard against careless errors! But it is a different problem from checking whether your software actually does the right thing.

The magnitude of change needed in unit tests while changing code is definitely maddening. For example: add an argument to a method. It's called only 3 places in production code, but each of those sites has 10 test cases, so there are 30 mock expectations to update now. And the IDE doesn't understand those so no automated support.

On the other hand, integration and end-to-end tests will go red when (seemingly) nothing at all has changed, because of drift in tangentially related components from other codebases. Usually the fixes are simple, but it requires a willingness to drop what you're doing and investigate when they happen. Otherwise they just get ignored.

Re: The testing pyramid should look more like a crab

#44

Post by closeparen »

Tainnor wrote: post #39
pydry wrote: post #38

Earlier quoted context omitted.

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense. However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree…

If I design tests to capture the spirit of the class / module, the line coverage is only 50-60%. In Go each method needs a lot of individual attention to exercise all the error return branches up to a 90%+ coverage standard.

Re: The testing pyramid should look more like a crab

#45

Post by pydry »

closeparen wrote: post #43
pydry wrote: post #38

Earlier quoted context omitted.

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

I think of it as double entry bookkeeping. You have to say what you mean two different ways, and those two phrasings have to be reconcilable. This is still a useful guard against careless errors! But it is a different problem from checking whether your software actually does the right thing. The magnitude of change needed in unit tests while changing code is definitely maddening. For example: add an argument to a met…

I think of it as writing everything twice to appease the unit testing gods. If done indiscriminately it really is more of a ritual to ward of the bug gremlins than an effective development practice.

Integration tests do suffer from a whole set of problems that unit tests do not but with one fundamental difference: they're usually problems that are soluble with better engineering techniques.

Integration tests that go red when something other than the code changes often provide useful feedback, so I wouldn't necessarily rule them out.

Re: The testing pyramid should look more like a crab

#46

Post by pydry »

Tainnor wrote: post #39
pydry wrote: post #38

Earlier quoted context omitted.

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense. However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree…

I agree. Good, stable abstractions covering complicated logic are where unit tests shine.

Unfortunately I find this scenario is also more the exception than the rule.

Big ball of mud apps with few effective tests are the norm and apps which do very little complex logic under the hood (focused instead mainly on hooking together services) are also not at all uncommon.

Re: The testing pyramid should look more like a crab

#47

Post by musingsole »

Tainnor wrote: post #41
musingsole wrote: post #15

Earlier quoted context omitted.

> end-to-end tests are slow as well as difficult to write+maintain Citation needed, sounds anecdotal. End-to-end tests should run as fast as any given use case (for most software, that means relatively quick), plus some set-up time as testing overhead. But that testing overhead is often self-inflicted and can be managed. I have anecdotes to prove it. For the speed to write end-to-end tests, shouldn't they be the easi…

I'm pretty sure more people care about the VAT being calculated correctly than about whether it's displayed on the left or on the right. You're taking a reasonable point of view - code matters only inasmuch as it has an effect on users - and take it to an absurd extreme.

Then you misunderstand. You are right that the VAT or similar financial matters require a very high degree of precision and accuracy. Physics simulations and similar computations even more so. These applications have a very LOW tolerance for error and -- rightfully so -- tend to have a lot of attention paid to their backends.

I contend that most applications are not these. I'll put it forward as professional experience that even financial adjacent systems have a surprising degree of error tolerance. Our engineers/scientists are constantly striving to improve correctness because that's what engineers do but it's plainly not a priority for our bottom line. Countless errors have been caught in the last year that made individual results quite questionable. But on the whole, the system and its results are accurate enough that we have plenty of customers and business keeping us busy.

Re: The testing pyramid should look more like a crab

#48

Post by pydry »

contravariant wrote: post #42
pydry wrote: post #18

Earlier quoted context omitted.

>If your code is well-structure You've alluded to the real benefit of end to end tests here, IMHO. They make very few demands about how your code is structured. This is fantastic if you've just been handed a big ball of mud. There is a definite trade off between faster running tests and tests which don't need to be rewritten every time you change some code.

Is that a benefit of end-to-end tests? Seems more like encouraging more unit tests would discourage bad coding habits if anything.

I don't find it does discourage bad coding habits. It just amplifies the problems they cause.

Re: The testing pyramid should look more like a crab

#49

Post by btilly »

CivBase wrote: post #36

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a ma…

I find that unit tests are much less useful than integration tests.

When we write unit tests, we ask the person to write the code to spend a lot of work documenting how they expect the system that they are integrating with to work. What happens when that system behaves differently? Or changes in the future? And the way that mocks are set up are very sensitive to implementation details, which is a maintenance burden when you have to change that.

As a simple example, I had a function that was loading data into a database with a series of insert statements. I sped it up by several orders of magnitude by using a COPY to send CSV data. Guess how well that worked with the mock database interface in the unit tests? (Long story short, it didn't. I replaced that unit test with an integration test.)

And yes, I've heard all of the lectures from test first folks. But somehow I've never had the experience of things working that way with my projects. (Possibly because most of what I do is close to the software/database interface.)

Re: The testing pyramid should look more like a crab

#50

Post by mpweiher »

btilly wrote: post #49
CivBase wrote: post #36

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a ma…

I find that unit tests are much less useful than integration tests. When we write unit tests, we ask the person to write the code to spend a lot of work documenting how they expect the system that they are integrating with to work. What happens when that system behaves differently? Or changes in the future? And the way that mocks are set up are very sensitive to implementation details, which is a maintenance burden w…

If it's talking to the database, it's not a unit test.

https://blog.metaobject.com/2014/05/why-i-don-mock.html

Post reply on HN