Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The testing pyramid is an outdated economic model

wiremock.io

Post reply on HN
Print view View original

41–45 of 45 posts

Re: The testing pyramid is an outdated economic model

#41

Post by bluGill »

I have long written the pyramid with the vertical axis of runtime. At the bottom are tests that run so fast you would run them every build even if there is no possible way your code changes could break those tests. Then tests that you run if they test what you just changed. Next tests you run on all CI builds [locally only when they fail]. Then tests you run regularly but not ever CI builds (every night or every week). Then tests that are so expensive you run them rarely - these generally are manual tests where a human is running the test in some real world condition that is hard to setup.

There is value in every level.

Re: The testing pyramid is an outdated economic model

#42

Post by mrkeen »

bluGill wrote: post #38
mrkeen wrote: post #15

We don't need this article because the 90%-unit/10%-itest was only ever a goal to aspire to. Just like achieving 90% code-coverage - no need for a thinkpiece to say that 40% or 60% is now 'the right amount' of code-coverage. We like units because they are fast, deterministic, parallelisable... all the good stuff. Relative to that ideal, integrations are slower, flakier, more sequential, etc. While I've never gone ful…

What is a unit? I have never seen a formal definition. Without that we cannot have any discussion. To some a unit is a function. To some it is a module (generally someone else's module). To some it is an entire application. To some it is the entire computer in your embedded device. To some it is the entire device... Most people have no clue what we are talking about and don't car (also should not care).

My working definition is that a unit is something which can be unit-tested. By that I mean: the code being tested has the same properties that you expect of a good unit-test. I guess fast and deterministic are good enough properties for a working definition.

writeFile(fname,"Hello, World") is only one thing, but its behavior will depend on the state of the filesystem, so it's not a unit.

parseComplicatedObject(bytes) could be a unit (even if it calls out to many other sub-parsers - as long as they are also units).

One thing I see in a lot of companies is efforts to reduce test flakiness. Devs will attempt this work in src/test. But if the code is flaky, and you change the test from flaky to reliable, then you have just decreased the realism of your test. Like I mentioned with my comment above, you reduce test flakiness by doing the hard work in src/main. The src/test changes should be easy after that.

Re: The testing pyramid is an outdated economic model

#43

Post by creesch »

hitchstory wrote: post #19
creesch wrote: post #7

Okay? This seems like a fluff blog post as the trophy concept was coined back in 2018 if I am correct. Coming from wiremock it makes sense given their product, but it is just marketing fluff. Honestly, as long as the GUI tip remains as small as possible I am mostly fine with whatever shape it takes below there. For modern web applications with a lot of APIs it does make sense to use a trophy. For other applications w…

I think the premise is correct and I think you are disagreeing with it. Yes, the pyramid was set out as a goal in its original incarnation. That was deeply wrong. The shape ought to be emergent and determined by the nature of the app being tested (i went into detail on what should determine that here https://news.ycombinator.com/item?id=42709404 ) Some of the most useful tests Ive worked with HAVE had a large GUI tip…

> Some of the most useful tests Ive worked with HAVE had a large GUI tip. The GUI behavior was the most stable surface whose behavior was clearly defined which everybody agreed upon.

This might be true, but that might also have said something about the layers below that and actually be a symptom for a larger issue within the development organisation.

> GUI tests are not inherently fragile or inherently too slow either.

Compared to testing APIs or Unit tests they are though. Not only do you need to navigate an interface with a machine that is actually intended for humans, you need to also deal with the additional overhead.

Re: The testing pyramid is an outdated economic model

#44

Post by giorgioz »

VMG wrote: post #30
giorgioz wrote: post #17

Earlier quoted context omitted.

> What AMD giveth, Electron taketh away. This is actually true but the moralistic negative tone and no explanation about it makes me think the writer did not understand why this is happening and why it has both PROs and CONs. It's similar to some other statements I heard before on this subject "It's pointless to add/increase roads, there will always be traffic". It's true there will always be traffic but it's not poi…

I didn't intend it as a negative tone. It is meant as an observation: while the raw system speed has increased orders of magnitudes, certain high-level operations seem to remain constant speed over time. The time it takes to booting an operating system, start a program, compile a program or run a test suite seems to remain somewhat constant over my career. It indicates that the determining factor is not the clock spe…

Thank you for answering and explaining @VMG! Yes exactly, some features of software will naturally gravitate to some values where they are good enough and something else becomes the bottleneck and determining factors. I still believe though there will be a niche space for people that need the extra performance because their use case takes an advantage from it.

Re: The testing pyramid is an outdated economic model

#45

Post by hitchstory »

creesch wrote: post #43
hitchstory wrote: post #19

Earlier quoted context omitted.

I think the premise is correct and I think you are disagreeing with it. Yes, the pyramid was set out as a goal in its original incarnation. That was deeply wrong. The shape ought to be emergent and determined by the nature of the app being tested (i went into detail on what should determine that here https://news.ycombinator.com/item?id=42709404 ) Some of the most useful tests Ive worked with HAVE had a large GUI tip…

> Some of the most useful tests Ive worked with HAVE had a large GUI tip. The GUI behavior was the most stable surface whose behavior was clearly defined which everybody agreed upon. This might be true, but that might also have said something about the layers below that and actually be a symptom for a larger issue within the development organisation. > GUI tests are not inherently fragile or inherently too slow eithe…

>something about the layers below

Absolutely. They were tests over a big ball of mud in a company I had joined recently.

This is, I think, the only good way to work with what is probably (unfortunately) the most common type of real world code architecture.

If your testing approach cant deal with big fragile balls of mud then it is bad. This is why I dont have a lot of respect for the crowd that thinks you must DI first "in order to be able to test". Such architectures are fragile and will break under attempts to introduce dependency inversion.

>Compared to testing APIs or Unit tests they are though.

In the above example there probably wasnt a single code interface or API under the hood that was any good. Coupling to any of those interfaces was fragile with a capital F if you actually expected to refactor any of them (which I did).

Even for decent quality code, the freedom to refactor interfaces is wildly underrated and it is curtailed by coupling a test to it.

Post reply on HN