Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The testing pyramid should look more like a crab

changelog.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: The testing pyramid should look more like a crab

#31

Post by Jtsummers »

ozim wrote: post #27
mannykannot wrote: post #16

Earlier quoted context omitted.

Unit tests don't just fail to find bugs because they don't give complete coverage (all testing has that problem, and the author is mistaken to say that end-to-end testing shows all bugs), they fail to find bugs resulting from faulty assumptions about how the units will interact. Boeing's faulty Starliner test demonstrated how that goes [1]. The true value in unit testing is in revealing the bugs it will find early in…

Pyramid approach, inverted pyramid, crab shaped does not make sense because it tries to be catch all guideline. Same with any 'maturity models' for software projects. People are forgetting there is something like 'risk based' approach to testing. This is the only way to allocate budget effectively, where stuff that can drop half of your database should be tested well and stuff that will make your UI look funny probab…

I hate to say this, but I do think CMMI's model is a good idea at its core. The model itself is actually pretty solid. That is, everything it describes is something a good organization should do, and the requirement to document the processes + train people is a sound one.

But the appraisals and how most businesses treat it are, well, moronic.

The appraisals are easily gamed so that if you do everything on paper you're pretty much going to get your target level (3 is good enough for 99% of contracts if matters at all). And businesses try to treat the model as a process, when it's not.

The former means that the appraisal results are useless. You cannot judge anyone by being level 3 or 5. That often just means they played the game better, not that they're actually better.

The latter, though, is the worst part. CMMI is broken into a number of process areas (I'm happy to report I no longer recall the number of areas). It describes them in psuedo-process or procedural terms. That is, you could take what they describe and make a process, but it would be a very linear and simple process. What's intended is for you to say how your process maps to their model.

But organizations don't do that, they do moronic things instead like scrap the above and write a whole new document that's 40 pages of nothing but plagiarizing the CMMI book saying how they do things, but it's almost entirely a lie because no one (except the sacrificial team sent to be appraised) actually does it that way.

Re: The testing pyramid should look more like a crab

#32

Post by jniedrauer »

lxe wrote: post #9

Don't do 'unit tests' at all. Your sweet spot in testing should be the fastest possible 'semi-integrated' (real clients, faked services) tests that can be done reproducibly and hermetically, testing specific end-goal functionality.

> real clients, faked services

I'm curious what you think this looks like in practice. I've had success with these kinds of tests, but only for very simple external services. For example, writing a fake elasticsearch backend for a few known inputs and outputs might be a good fit. But what about an RDBMS? You can't realistically write a mock backend that can talk to a real client.

Re: The testing pyramid should look more like a crab

#33

Post by jniedrauer »

ozim wrote: post #24
lxe wrote: post #9

Don't do 'unit tests' at all. Your sweet spot in testing should be the fastest possible 'semi-integrated' (real clients, faked services) tests that can be done reproducibly and hermetically, testing specific end-goal functionality.

I would agree because I see my devs writing unit tests for plumbing code that does not have any logic. I am annoyed by that. So I am 'party pooper' when it comes to unit testing. But I would like to have unit tests for something that has calculations or non trivial logic only. Other thing I would really like 'risk based' approach, like what is the worst thing that can happen with that code. If it can drop half of the…

There is some value in unit tests for plumbing. Something I see a lot is a set of values being transformed across several domains, losing data at each step. A very simple test that inputs values on one end of the pipe, and checks the values on the other end of the pipe, makes it very easy to be sure that you didn't drop something.

Re: The testing pyramid should look more like a crab

#34

Post by dpix »

tmcneal wrote: post #29

I'm co-founder of an end-to-end testing product, but I don't actually agree that better end-to-end testing coverage means you should create less unit tests. The reason why is because there's a class of things you want to test that are much better served at the unit testing level. Unit tests are always going to be faster to execute and by definition have a much smaller scope, which is helpful in reasoning about what y…

The author talks about how many other sources of bug there might be outside of your code like integration, assumptions, compilers etc that e2e tests will help catch.

Well at least a large portion of those bugs might be able to be caught by unit tests! No reason to write less of those, having good unit and integration tests still make you more confident about your application. And because they are faster to run the feedback loop when they break is also faster

Re: The testing pyramid should look more like a crab

#35

Post by lxe »

jniedrauer wrote: post #32
lxe wrote: post #9

Don't do 'unit tests' at all. Your sweet spot in testing should be the fastest possible 'semi-integrated' (real clients, faked services) tests that can be done reproducibly and hermetically, testing specific end-goal functionality.

> real clients, faked services I'm curious what you think this looks like in practice. I've had success with these kinds of tests, but only for very simple external services. For example, writing a fake elasticsearch backend for a few known inputs and outputs might be a good fit. But what about an RDBMS? You can't realistically write a mock backend that can talk to a real client.

As much as it's reasonable, of course. If it's impossible to spin up a mock service or a DB, then client fakes can be used at the cost of narrowing the test's coverage and lowering authoritativeness.

Re: The testing pyramid should look more like a crab

#36

Post by CivBase »

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes.

Of course, just because each individual part of a machine works, it doesn't mean the machine itself works. That's where integration tests come in. Turns out it's usually fairly easy to make sure all the parts work together once you've verified all of the parts individually, so you usually need fewer integration tests.

Once you've verified all the parts work and that they work together correctly, now all you need to do is make sure you actually made the right thing. Enter functional tests and E2E tests. These are usually especially costly to maintain, but they're also so high-level that you usually need very few of them.

Writing tests is easy, but they're only useful if you run them. It's not enough for a test to work when you write it - it has to continue working into the future. Maintenance is a recurring cost and, in my experience, is where automated testing can get really expensive. The testing pyramid helps minimize that cost.

In an ideal world, you would maximize testing at every level - but in the real world that's often just too expensive.

Re: The testing pyramid should look more like a crab

#37

Post by jschwartzi »

username90 wrote: post #14
r0s wrote: post #8

End to end tests are less precise to triage when they break, and they break more often than unit tests. As the author said, they show all bugs, so they break a lot. When the test is broken, it's not providing coverage until it's fixed. Which is why I advocate for less assertive testing in that case: https://assertless.org/ The priority of fixing end to end tests becomes critical, and so test maintenance is more impor…

> and they break more often than unit tests. That isn't my experience, unit tests breaks all the time and has to be rewritten from simple refactoring while larger tests pass as long as you keep behavior the same. This means that the bigger tests are way better for refactoring and safely making changes to your code. If most of your tests are unit tests and you don't have bigger tests covering the same things then refa…

If your unit tests are breaking all the time, try testing less at the unit level and more at the integration level. There are a lot of dogmatists out there who preach that unit tests should have "X % coverage" no matter what, but that's a ridiculous claim. Different projects will require different mixes of testing strategy, and part of the engineering process is arriving at your verification and validation methods, codifying them, ensuring that they provide valid output, and making sure you're using your time efficiently. If most of your time is spent fixing tests, re-evaluate whether those tests are worth doing that way. Your time might be better spent testing at a higher level or even manually.

Re: The testing pyramid should look more like a crab

#38

Post by pydry »

CivBase wrote: post #36

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a ma…

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring.

The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

Re: The testing pyramid should look more like a crab

#39

Post by Tainnor »

pydry wrote: post #38
CivBase wrote: post #36

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a ma…

Unit tests are not intrinsically easier to maintain. The lower level the test the more it reflects implementation rather than specification and hence the more changes it demands when refactoring. The practical upshot of these types of tests is often that they catch very few actual bugs but nonetheless demand constant and expensive maintenance when code is changed.

This is true if your units under tests are bad abstractions. If you don't know yet what you will build, if your requirements change all the time, good abstractions maybe rarely emerge and then maybe unit tests make less sense.

However, I've worked on a number of applications where there were very well-defined units that were quite stable in terms of their abstraction, and then unit tests are a joy. "I have this tree structure and I need to transform it into another structure" and stuff like that.

It's also worth keeping in mind that "unit testing" doesn't have to mean "class (or method) testing". Sometimes the class is the right level of abstraction but sometimes it can also be a small module of classes working in concert etc.

I agree that tests shouldn't simply reflect implementation, but there are functional requirements that can be specified at a lower level.

Re: The testing pyramid should look more like a crab

#40

Post by Tainnor »

CivBase wrote: post #36

This article seems to fundamentally misunderstand why the testing pyramid is a pyramid in the first place. You shouldn't have more unit tests because they're easier to write. You should have more unit tests because they're easier to maintain. By definition, they focus on a small part of the software and should not break if an unrelated part of the software changes. Of course, just because each individual part of a ma…

It's not just that the article misunderstands that unit tests are easier to maintain (if well written at least), but they also don't seem to discuss the problem of combinatorial explosion that plagues end-to-end tests.

There are only two ways of avoiding that problem: ignoring it and simply give up on comprehensive testing, or realise that comprehensive testing can only be done by writing mostly unit tests.

Post reply on HN