We killed our end-to-end test suite • Hacker News Board

Re: We killed our end-to-end test suite

#161

Post by Jtsummers » Fri, Sep 24, 2021, 8:58 PM UTC

jrochkind1 wrote: post #153

amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have. What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?

Integration tests may integrate smaller-than-the-whole groups of subsystems. It definitely gets fuzzy. A lot of people treat end-to-end and integration tests as equivalent, but piecing together everything-but-the-frontend and testing it is also an integration test, but not an end-to-end test.

If we consider tests as existing at and covering different scales, unit tests are at the smallest scale and integration tests run the gamut from 2 units to the entire system.

Re: We killed our end-to-end test suite

#162

Post by lupire » Fri, Sep 24, 2021, 9:00 PM UTC

gravypod wrote: post #151

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

One of the things that I love about Bazel is it thinks of a binary that obeys a contract as a test. This means you can have things like `sh_test` which just runs a shell script in a sandbox and gives you all of the benefits Bazel has normally for test execution. You get automated caching, parallelization, and remote execution of tests for free. A great talk about this: https://www.youtube.com/watch?v=muvU1DYrY0w You…

> it thinks of a binary that obeys a contract as a test.

who doesn't?

Re: We killed our end-to-end test suite

#163

Post by Jtsummers » Fri, Sep 24, 2021, 9:00 PM UTC

aidenn0 wrote: post #155

Jtsummers wrote: post #93

Earlier quoted context omitted.

Not GP, and fortunately not often, but I have seen that done to overcome race conditions. I pushed for it to be corrected by using a proper design. That was a stupidly hard fight, though.

My pet peeve is people sprinkling C's "volatile" keyword in places. Since doing so inhibits many optimizations, it changes the timing and can make race conditions appear to go away.

Yep. Lots of effective ways to paper over issues without actually resolving them, and often disguising them so that resolution becomes nearly impossible later.

Worst, things like the introduced sleeps in some of the systems look legit. There are reasonable times to introduce a timed delay into your program (3rd party APIs have a rate limit, 1 request per second or 10 per 30 seconds or whatever). Depending on how you introduce these extra sleeps, then, it's possible that they'll look like they satisfy a valid requirement, when the reality is that they exist to cover up the absence of things like proper use of locks/mutexes or other elements.

Re: We killed our end-to-end test suite

#164

Post by _hilro » Fri, Sep 24, 2021, 9:03 PM UTC

mirekrusin wrote: post #129

So they ditched e2e in favour of something that average monorepo checks statically ie throught typescript? Then mocked functional tests and called it a day? The problem probably started when they put themselves in this microservice plague setup where they can't spawn simulated environment in ci anymore. As it turns out running system aka deployment on environment is a monolithic expression of microservice spaghetti.…

> As a side note flaky tests is such an idiotic concept. There are tests that pass and ones that don't. How good is button which works 35% of time? It's not a good button, period

Or more like the test runner succumbs to non deterministic flaky behaviour.

If something failed 65% of the time, it would be one of the easiest thing in the world to fix.

If it fails .001% of the time, that's what the industry refers to as flaky.

> Forbit flakiness, there is no such thing as passed flaky test - those are just shitty tests.

Have you ever written and monitored e2e tests over a year? It's industry wide.

Selenium/selenium grid always works great until it doesn't. Ditto with the new kids on the block. e2e outside of a browser is 100% fine unless there's an actual bug somewhere.

Re: We killed our end-to-end test suite

#165

Post by asdfasgasdgasdg » Fri, Sep 24, 2021, 9:04 PM UTC

afiori wrote: post #115

asdfasgasdgasdg wrote: post #21

Earlier quoted context omitted.

I can hardly think of a situation where I'd want no end to end test. I think one misconception is that there has to be a single end to end test. Really what you want is a variety of end to end tests examining the functionality of different parts of the system. But the system under test is still the whole system, not the units. These partial end to end tests can still be quick to run, as long as you keep system startu…

there was a nice talk putting this as an example where React's engineering teams designed their test to be future proof: always test the public API; your end to end tests should survive a major refactoring/rewrite

Hard to overstate the benefit of not having to rewrite your tests when you refactor. That's a concrete and difficult to ignore benefit of using higher level APIs. I find that the cost of writing a test is often not less than the cost of writing the code. If that is correct, it means that the system lifetime cost of authorship is much higher with unit tests. If all tests are unit tests the cost is almost 100% higher.

Re: We killed our end-to-end test suite

#166

Post by SquishyPanda23 » Fri, Sep 24, 2021, 9:05 PM UTC

aidenn0 wrote: post #38

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…

> Flaky tests are an indication of non-determinism either in your test or your system

Or in the system that runs your tests. That can itself be non-trivial.

Re: We killed our end-to-end test suite

#167

Post by mattnewton » Fri, Sep 24, 2021, 9:05 PM UTC

grumple wrote: post #156

mattnewton wrote: post #149

Earlier quoted context omitted.

I should be able to test that this usually works though, right?

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are pr…

Right, but the key word here being “usually” - if I can’t just run the test three times and assume 2/3rds of the time it’s good, how can I know it usually works in production?

Is the right solution really to throw up your hands and not test end to end ever? I guess the argument is more convincing if it’s not that it’s impractical, it’s just too expensive relative to the returns.

Re: We killed our end-to-end test suite

#168

Post by lupire » Fri, Sep 24, 2021, 9:06 PM UTC

hinkley wrote: post #124

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

Going the other way, unit tests prevent devs from writing systems that behave correctly.

It doesn't help to prove that a module does what you expect, if your expectations are wrong.

Re: We killed our end-to-end test suite

#169

Post by lamontcg » Fri, Sep 24, 2021, 9:10 PM UTC

mst wrote: post #143

gls2ro wrote: post #113

Earlier quoted context omitted.

Ok. In this case it is still e2e. In my head e2e is never a strategy to test everything. I also thing they did a good optimizations to offload some cases from e2e go some other form of testing which for should be integration testing. If they dont do integration testing then a lot of possible bug cannot be found. Just testing the input and output of each microservice is not enough. But I will watch the video as maybe…

Yes, I said it was still e2e. What they meant - I think - was they moved away from "a primarily end-to-end integration test suite" to "service-as-black-box unit-ish testing" plus "focused end-to-end integration testing for critical paths".

Yeah and that's deeply confusing.

At work we have too many components that are tested in isolation, but which have grown to become tightly coupled, so we're trying to build an end to end testing framework.

So from my perspective I'm living in a world where our end-to-end test suite doesn't exist and therefore could be equivalent to "killing it" and it is bad. Each component tests its own contracts, but if there's no global testing that the contracts match in both codebases then you're still shipping broken software.

I thought this article would be some clever way to match client side and server side contracts to ensure that the contracts are identical on both sides and tested so that you could test in isolation then still come away with assurances that the whole would work together.

Instead it sounds like it is advice to only build as much end-to-end tests so that you're reasonably confident that more isolating unit/functional will work, but don't build too much because they're horribly slow, and never adopt a policy that literally everything should be end-to-end tested because that will result in infinitely long running test suites. If you have no end-to-end tests you have no confidence that the software you ship works, if you have only end-to-end tests you have no confidence in your ability to ship software in the future.

So, uh, "clickbait title" I guess is my point?

Re: We killed our end-to-end test suite

#170

Post by BeetleB » Fri, Sep 24, 2021, 9:20 PM UTC

hinkley wrote: post #124

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

While I agree that writing unit tests are a lot harder, and you develop good skills in attempting to write them, I must say that in the projects I worked on most bugs were caught by integration tests (technically not E2E tests), and not unit tests.

I've also had projects with only unit tests, and almost no bugs were found by it, and there were plenty of bugs.

Ideally, I would like both. But if I had to have only one, I'd go with tests at a coarser granularity than unit tests.