Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

111–120 of 270 posts

Re: We killed our end-to-end test suite

#111

Post by xrd »

I'm a big advocate for testing, to state up front.

E2E is problematic from the start because of the expectations set by the name. Any sufficiently interesting system is nigh impossible to test "end to end." And, you aren't testing ends, you are testing the "start" of the process, to one of many "ends."

What about only doing "end testing?" Meaning, don't test the beginning. Put unit tests there. Put integration tests between the important components.

It is important to make sure you have coverage with automated tests that prove that your system can work at the end of at least some of the processes. Otherwise your QA costs are massive, and that never scales, and no one will ever fix that other than adding QA. Your innovation will slow to a crawl, much worse than waiting on your test suite.

I'm not sure after reading this article that the authors added a new testing methodology by calling it "contract testing." I'm still confused about what that means. Having said that, I am still confused about a lot of the boundaries between e2e and integration. It always sounds simple, but rarely in practice.

The bottom line: the organization as a whole has to see the value of testing. That's harder work than writing the tests for sure.

Re: We killed our end-to-end test suite

#112

Post by arp242 »

3pt14159 wrote: post #96
deckard1 wrote: post #70

Earlier quoted context omitted.

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in additio…

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).

Pretty much this. I found that having loads of E2E tests often doesn't add all that much; usually they're all doing the exact same thing test after test after test, and since these parts tend to be fairly isolated there isn't all that much that can go wrong in just that specific test. Either it works for everything, or it fails for everything.

The way I've always viewed E2E tests is as "testing everything at the top layers" such as middleware and whatnot, which you can usually do with just a few (or sometimes even one) test. Other less high-level integration tests can test all the rest, and they tend to run much faster as they avoid a lot of overhead, and are a lot easier to write and reason about, especially if tests fail.

I once rewrote a E2E test suite to use integration tests which gave a massive speed-up, and because the tests were a lot easier work with people actually started writing them. I added a few E2E tests (IIRC logging in, viewing the dashboard, logging out) and that was enough really.

Re: We killed our end-to-end test suite

#113

Post by gls2ro »

mst wrote: post #109
gls2ro wrote: post #84

Earlier quoted context omitted.

I cannot watch this entire video, so it is possible that you explain conceptually how your acceptance tests are different from end to end test? I would like to undestand what were the goals of your initial end to end and what are the goals of acceptance testing, and how you define these acceptance testing from test objective perspective. I assume end to end tests could be described by this definition: "test the funct…

I think what they mean, basically, is "e2e testing absolutely everything was becoming a nightmare, so we've now switched to 'contract based testing' - effectively 'unit testing where the microservice is the unit granularity' - plus some e2e style testing for critical paths where it's still valuable enough to justify all the extra effort".

Ok. In this case it is still e2e. In my head e2e is never a strategy to test everything.

I also thing they did a good optimizations to offload some cases from e2e go some other form of testing which for should be integration testing.

If they dont do integration testing then a lot of possible bug cannot be found. Just testing the input and output of each microservice is not enough.

But I will watch the video as maybe it is better explained there and there is something to learn from this experience.

Re: We killed our end-to-end test suite

#114

Post by marcosdumay »

munificent wrote: post #75
kodablah wrote: post #33

Earlier quoted context omitted.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

> is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code. Another way to say this is that efficient e2e tests require significant continuous investment in top-tier engineer time. The question then is how much engineering time is worth being spent in that way. It may be that, yes, you can h…

> The question then is how much engineering time is worth being spent in that way.

Well, since it brings more value than testing in a lower level, I would say, more than any other kind of test (except, maybe, for monitoring).

Another good question is, is there any kind of tests that gives you good results without investing good-engineers time? If you find any, I'd ask you to share (but I would understand if you consider the information a market differentiator and won't).

Re: We killed our end-to-end test suite

#115

Post by afiori »

asdfasgasdgasdg wrote: post #21
msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

I can hardly think of a situation where I'd want no end to end test. I think one misconception is that there has to be a single end to end test. Really what you want is a variety of end to end tests examining the functionality of different parts of the system. But the system under test is still the whole system, not the units. These partial end to end tests can still be quick to run, as long as you keep system startu…

there was a nice talk putting this as an example where React's engineering teams designed their test to be future proof: always test the public API; your end to end tests should survive a major refactoring/rewrite

Re: We killed our end-to-end test suite

#116

Post by ridaj »

Erm most people need all kinds of tests, it's not a story of one vs the other... Unit, integration, end to end... The end to end is always slowest and flaky so not everything should go there but with any mature ish product it's also impossible to catch issues related to system complexity without it... It's also weird how self important this account reads. The field is not new but they don't really describe how their homegrown stuff does better than other frameworks and present themselves as visionaries... Smelly

Re: We killed our end-to-end test suite

#117

Post by sidlls »

derefr wrote: post #82
aidenn0 wrote: post #38

Earlier quoted context omitted.

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

Third-party services should be mocked for integration and end-to-end testing. Error conditions with respect to these services should be something that is monitored and alerted on when appropriate.

Re: We killed our end-to-end test suite

#118

Post by evandwight »

kodablah wrote: post #33
Daishiman wrote: post #26

Earlier quoted context omitted.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

How do you run tests in parallel if part of the logic you are testing is a sql statement?

Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.

Re: We killed our end-to-end test suite

#120

Post by lrem »

Daishiman wrote: post #26
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

Of course they're the slowest and flakiest - they include the most sources of slowness and flakiness that you have. But, if you want to make your production fast and reliable, they're pretty good gauges if you're headed for success.
Post reply on HN