Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The testing pyramid should look more like a crab

changelog.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: The testing pyramid should look more like a crab

#51

Post by username90 »

jschwartzi wrote: post #37
username90 wrote: post #14

Earlier quoted context omitted.

> and they break more often than unit tests. That isn't my experience, unit tests breaks all the time and has to be rewritten from simple refactoring while larger tests pass as long as you keep behavior the same. This means that the bigger tests are way better for refactoring and safely making changes to your code. If most of your tests are unit tests and you don't have bigger tests covering the same things then refa…

If your unit tests are breaking all the time, try testing less at the unit level and more at the integration level. There are a lot of dogmatists out there who preach that unit tests should have "X % coverage" no matter what, but that's a ridiculous claim. Different projects will require different mixes of testing strategy, and part of the engineering process is arriving at your verification and validation methods, c…

Right, I never write traditional unit tests, I just test the API boundaries as you said. Sometimes it makes sense to draw a new API boundary in the code for some complex logic that is unlikely to change, but most of my tests looks pretty close to others integration tests.

Re: The testing pyramid should look more like a crab

#52

Post by vosper »

mcv wrote: post #3

> "And when we look at what Cypress allows you to do (which is write many useful tests that have very little flake) then you wanna write more end-to-end tests." Very little flake? That's not my experience with Cypress. The Cypress e2e tests on our build server sometimes fail for no discernible reason. Then you run them locally, and different tests fail, sometimes for understandable reasons (but then why didn't they f…

> The Cypress e2e tests on our build server sometimes fail for no discernible reason.

You'll probably be interested to hear that Cypress are... uhm... addressing this by adding "flaky test detection" and some amount of automatic retrying in v5.

We use Cypress at work, we have flaky test problems, but I'm not convinced much of it is actually Cypress's fault. As others have mentioned, writing front-end code to be testable by E2E is not necessarily straightforward, and when you're trying to add tests to a big existing codebase I think you kinda have to accept some flakiness and retrying.

What I've been wondering about recently though is: if you construct applications in a certain way do you still need real E2E tests? For example, React is declarative, and with a state-management system like Redux shouldn't you be able to get the same kind of assurance that E2E gets you without needing an actual browser? Maybe things like animations, visibility etc are where that comes unstuck?

Re: The testing pyramid should look more like a crab

#53

Post by alkonaut »

Tests should document behavior and aid in refactoring and maintaining boundaries. E2e tests don’t document modules (functions/types), only the highest interface, and it doesn’t maintain any separation of layers etc.

All kinds of tests are needed, but I find the pyramid arises from the fact that there are a lot more low level units than high level behaviors.

Re: The testing pyramid should look more like a crab

#54

Post by username90 »

hinkley wrote: post #21
hinkley wrote: post #20

By moving the 'meat' of the tests up high into the hierarchy, the author has just re-invented the testing ice cream cone with a different flavor. In software, we can make pretty much any process work for 18 months, before it starts to fall apart. If you don't stay at a company for at least a couple of years after they start doing testing in earnest, you won't really see that what you're doing doesn't scale/isn't resi…

This got so long I decided to split it in two: It is important to note you can't pull all of the tests down. I tend to use a plumbing analogy that I probably have stolen from the literature or a mentor - It's likely that QA already tested all of the pipes at the factory before they were installed in your house. But every plumber will turn on the taps and flush the toilet at once before they leave, to make sure the st…

> If the first person writes elaborate mocks instead of simplifying the design to not need them, they've locked in the implementation in a way that unit tests don't (Sunk Cost). This can turn a masochistic programmer into a sadist.

You mostly use elaborate mocks for unit tests. There is much less need for mocking in integration tests.

Re: The testing pyramid should look more like a crab

#56

Post by Dylanlacey »

Vinnl wrote: post #12
mcv wrote: post #3

> "And when we look at what Cypress allows you to do (which is write many useful tests that have very little flake) then you wanna write more end-to-end tests." Very little flake? That's not my experience with Cypress. The Cypress e2e tests on our build server sometimes fail for no discernible reason. Then you run them locally, and different tests fail, sometimes for understandable reasons (but then why didn't they f…

As I remember it Protractor was just a wrapper around WebDriver with some glue code to wait for Angular doing its thing after data changes. If you're using e.g. React or anything else that does not do dirty checking, you should still be able to use WebDriver. (That said, that's certainly still flaky enough.)

The cool part was that all of the extra JS Protractor used (sometimes for native WebDriver tasks) made things _less_ reliable.

Re: The testing pyramid should look more like a crab

#57

Post by contravariant »

pydry wrote: post #48
contravariant wrote: post #42

Earlier quoted context omitted.

Is that a benefit of end-to-end tests? Seems more like encouraging more unit tests would discourage bad coding habits if anything.

I don't find it does discourage bad coding habits. It just amplifies the problems they cause.

Well I suppose any form of discouragement is pointless if it is not recognised and acted upon.

Re: The testing pyramid should look more like a crab

#58

Post by btilly »

mpweiher wrote: post #50
btilly wrote: post #49

Earlier quoted context omitted.

I find that unit tests are much less useful than integration tests. When we write unit tests, we ask the person to write the code to spend a lot of work documenting how they expect the system that they are integrating with to work. What happens when that system behaves differently? Or changes in the future? And the way that mocks are set up are very sensitive to implementation details, which is a maintenance burden w…

If it's talking to the database, it's not a unit test. https://blog.metaobject.com/2014/05/why-i-don-mock.html

I'm aware of this.

If it is talking to a database, it's not a unit test.

If you've mocked out the database, it is a unit test.

My point is that the unit test version is a lot of work to create and tends to be useless. (Which is similar to the point that your blog link made.)

Re: The testing pyramid should look more like a crab

#59

Post by mpweiher »

btilly wrote: post #58
mpweiher wrote: post #50

Earlier quoted context omitted.

If it's talking to the database, it's not a unit test. https://blog.metaobject.com/2014/05/why-i-don-mock.html

I'm aware of this. If it is talking to a database, it's not a unit test. If you've mocked out the database, it is a unit test. My point is that the unit test version is a lot of work to create and tends to be useless. (Which is similar to the point that your blog link made.)

No. Stubbing out the database does not turn a test into a unit test , it turns it into a stubbed-out integration test. (Mocking is something different)

And since the point of an integration test is to test the integration, having a stubbed out version of that is fairly pointless, as you have discovered.

If you create actual unit tests, and allow those to drive your design, you will find much more joy.

Re: The testing pyramid should look more like a crab

#60

Post by nullsense »

contravariant wrote: post #42
pydry wrote: post #18

Earlier quoted context omitted.

>If your code is well-structure You've alluded to the real benefit of end to end tests here, IMHO. They make very few demands about how your code is structured. This is fantastic if you've just been handed a big ball of mud. There is a definite trade off between faster running tests and tests which don't need to be rewritten every time you change some code.

Is that a benefit of end-to-end tests? Seems more like encouraging more unit tests would discourage bad coding habits if anything.

As always in these discussions I like to remind everyone that you can have good and bad implementations of everything, and just doing unit testing doesn't make your engineers suddenly understand how to write testable code. So, you can well end up in a situation with a reasonable amount of code coverage, but not quite as high as you might like and when someone invariably digs into why they find while parts of the system are testable, parts of the system aren't. Now you're left with a large test suite that provides marginal value because it doesn't give you confidence your code can be deployed if just because it's passing, but the shite engineering practices heavily test implementation details which make maintenance slow to a crawl.

Good engineering is what creates good engineering. What you actually do, I suspect matters a lot less.

Post reply on HN