We killed our end-to-end test suite • Hacker News Board

Re: We killed our end-to-end test suite

#231

Post by Too » Sat, Sep 25, 2021, 5:50 AM UTC

strken wrote: post #203

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I've been thinking about the benefits of only writing E2E smoke tests which cover a small number of critical paths quickly. Seems like most of their problems came about because they wrote more tests than they needed, with higher coverage than was necessary.

If you are only allowed to have a single test in your project, it would have to be the E2E smoke. I've seen systems where master don't even start up for weeks but project is still proud to present unit tests are green with 100% coverage. One single E2E smoke outweighs all those tests, catching everything from faulty configuration, infrastructure, interface assumptions, integrations, libraries and code bugs. After this it becomes more of a runtime vs coverage balance.

What you need to be aware of is that even one single E2E test can require a significant investment in bootstrapping your test-environment. If you are clever you will use this to improve the quality of your production code as well. For example, if you only have a single DB for production and now need to spin up an new instance for test, don't do it manually, instead refactor it into infrastructure-as-code and you've now turned this component into cattle instead of sheep, allowing you to scale both prod and test and giving your ops team a much easier life.

Re: We killed our end-to-end test suite

#232

Post by Too » Sat, Sep 25, 2021, 6:06 AM UTC

ahuth wrote: post #128

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> If you maintain speed and strict curation of such a suite This seems to be the hard part. Any tips for maintaining speed and strict curation, especially at scale (in terms of developers)? At the very least, it seems that E2E tests are a tool that's easy to misuse. Not sure of the best way to mistake-proof it.

Automatic linter completely banning sleep(). Present reasonable alternative functions instead, like untilServiceAvailable.

Sounds silly and obvious but look into your test suite and i'm sure you will find sleep everywhere.

Majority of all flaky tests are due to miscalculated sleeps in my experience. Sleep is also the biggest contributor to slow runs, with longer and longer sleeps being used to combat the flakiness. Both factors being the biggest pain points of E2E tests.

Re: We killed our end-to-end test suite

#233

Post by mirekrusin » Sat, Sep 25, 2021, 6:38 AM UTC

_hilro wrote: post #164

mirekrusin wrote: post #129

So they ditched e2e in favour of something that average monorepo checks statically ie throught typescript? Then mocked functional tests and called it a day? The problem probably started when they put themselves in this microservice plague setup where they can't spawn simulated environment in ci anymore. As it turns out running system aka deployment on environment is a monolithic expression of microservice spaghetti.…

> As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a good button, period Or more like the test runner succumbs to non deterministic flaky behaviour. If something failed 65% of the time, it would be one of the easiest thing in the world to fix. If it fails .001% of the time, that's what the industry refers to as f…

Let me clarify, I think I haven't expressed myself well. What I meant to bash is the culture where you wrap e2e tests with retries and consider flaky tests green. This arrangement is a pathology.

Industry did not put 1 out of 100000 (.001%) or less as threshold for calling something flaky. From my experience I've seen 20% success rate tests passing due to retries and teams are living with it as normal.

> Have you ever written and monitored e2e tests over a year? It's industry wide.

Yes, on high profile projects. I find myself repeating how important determinism is. Flaky tests, no mater how frequent, are indistinguishable from bugs, which implies they can't ever be considered green. Architecting testing environment in such a way that it is deterministic is fundamental. Sometimes it doesn't require big reshuffles, it just means the test has to be rephrased in deterministic terms that matter without asserting intermediate, timing based, racing middle states. As an example testing random failures by killing services doesn't have to assert intermediate client states, it has to assert that final state is eventually correct, which implies reconnects did happen and eventually state is correct, regardless of intermediate client states (ie. retry logic on the client auto healing itself on idempotent actions or erroring notifying client the service is offline, in which case user itself has to retry action - both are ok depending on how long service was offline, both can be progressed from test PoV, assertion that specific one happened is irrelevant).

Re: We killed our end-to-end test suite

#234

Post by arp242 » Sat, Sep 25, 2021, 6:50 AM UTC

tetha wrote: post #133

3pt14159 wrote: post #96

Earlier quoted context omitted.

In my book, E2E tests should be on a couple of basic, mission critical things and integration tests should pick up the rest. It's far, far better to have 10 E2E tests and 1000 integration tests than 0 E2E tests and 1500 integration tests because it picks up failures in your infrastructure or weird stuff like middleware that are probably system wide(ish).

I feel like this is also where dogfooding - or drinking your own champagne - comes in, if possible. We can use our software internally and sure, there are hardware costs and manpower overheads to run an additional instance of our software, but those aren't too high. Hardware necessary to run E2E tests of all the systems at proper scale including maintenance manpower probably eclipses those efforts. And then you'd hav…

This is absolutely a good attitude, but it only works for a fairly limited class of software. I worked for things like realtor agent software, child care agency software, etc. and you can't really "dogfood" these sort of systems unless you want to become a landlord or start a childcare service.

Re: We killed our end-to-end test suite

#235

Post by sumedh » Sat, Sep 25, 2021, 7:11 AM UTC

amw-zero wrote: post #146

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

> At least for web applications, all end to end test suites are slow and flaky.

Then you should be asking why are they flaky in the test environment? Its probably because your services are running on very slow servers.

Re: We killed our end-to-end test suite

#236

Post by mtone » Sat, Sep 25, 2021, 7:31 AM UTC

evandwight wrote: post #118

kodablah wrote: post #33

Earlier quoted context omitted.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

How do you run tests in parallel if part of the logic you are testing is a sql statement? Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.

Here our tests are written in BDD (behavior-driven development) style, mimicking user actions and data expectations. During development, these are run against mocks (either in-memory DB or a mock repository). Individual small scenarios are also combined into realistic long-running processes, for example cases from opened to closed taking various paths.

The suite runs in parallel fast and frequently alongside unit tests. Then occasionally like before PR merges, the same scenarios are run against a clone of the production environment to catch any mismatch with the run-time environment, also in parallel (connections) to simulate multi-user usage. Any technical issue prompts an improvement of the mocks and rarely resurfaces.

Running these as scripts also doubles as a fake data generator to play with for manual testing, reporting, etc. Last we proceed with some manual testing to validate new changes and pick-up UI-related issues - we don't do UI automation.

Re: We killed our end-to-end test suite

#237

Post by jseban » Sat, Sep 25, 2021, 10:35 AM UTC

williamdclt wrote: post #177

jseban wrote: post #152

Earlier quoted context omitted.

Yeah but what's the point then, in that case you can just take back the old QA team and delete the gazillion lines of e2e test code and save yourself the liability of all that complexity. If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard?

> If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard? Because you don't want to make _a_ manual test, you want to make _hundreds_ of tests.

If you are a human you have judgement and can determine which tests are the most critical and most relevant, so you don't have to always execute all of them. "It's just one line of css change to fix the styling, ok deploy". Second, a team of QA can very well make hundreds of tests in a day. And more importantly, they can really easily make decisions and draw conclusions such as "it's a bit slow sometimes, but overall acceptable", or "the animation is displayed correctly" or "there was a glitch in the rendering, but it's fine now", or "it works but the stying has moved slightly off center" etc etc, which take test expert programmers forever to try, and fail, to create deterministic automated tests for.

Re: We killed our end-to-end test suite

#238

Post by continuational » Sat, Sep 25, 2021, 10:40 AM UTC

fndex wrote: post #106

chartpath wrote: post #77

Earlier quoted context omitted.

Not in all environments, e.g. clojure.spec, or with MyPy reflection etc.

clojure.spec is not a type system. Still, my point is, just using a typed language won't remove the need for contracts, you would still need to roll up something like Nubank did even if it means using MyPy reflection features. For example, imagine you have two services that communicates through a message queue. Service A produces X as a string, but Service B consumes X as an integer. You can type that, both services…

Hm? Give the message a type. Service A or B would fail to compile, depending on whether the static type of X is a string or an integer.

Re: We killed our end-to-end test suite

#239

Post by kdazzle » Sat, Sep 25, 2021, 12:24 PM UTC

amw-zero wrote: post #145

derefr wrote: post #82

Earlier quoted context omitted.

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)

There's basically no value in having tests against third-party code anyway, because all the test is going to do is tell you that they broke their interface. And by then, production is already broken.

I don’t know, I think having your tests assume that the 3rd party data looks a certain way is helpful. If that ever breaks in prod, then something needs to change, and your tests can change if the interface changes.

Re: We killed our end-to-end test suite

#240

Post by hinkley » Sat, Sep 25, 2021, 3:55 PM UTC

mbrodersen wrote: post #217

hinkley wrote: post #124

Earlier quoted context omitted.

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

Unit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.

You can’t test all error cases end to end. If you can you have shitty error handling.

Clock skew between servers? Drifting clock skew? Disk space exhaustion? Disk space exhaustion at each possible failure point? There are so many of these and you’re going to inject most of them in unit tests.

My original point was that if you can’t write good unit tests your e2e tests are also going to be lousy, and you will never get good at either, let alone both, if you fixate on more coverage with E2E tests.

They’re also just too damned expensive even if they were qualitatively as good. Which they are not. They are less numerous, sure, but that’s false economy because they are usually 3 orders of magnitude slower.