We killed our end-to-end test suite • Hacker News Board

Re: We killed our end-to-end test suite

#201

Post by alex-freire » Sat, Sep 25, 2021, 1:17 AM UTC

lamontcg wrote: post #169

mst wrote: post #143

Earlier quoted context omitted.

Yes, I said it was still e2e. What they meant - I think - was they moved away from "a primarily end-to-end integration test suite" to "service-as-black-box unit-ish testing" plus "focused end-to-end integration testing for critical paths".

Yeah and that's deeply confusing. At work we have too many components that are tested in isolation, but which have grown to become tightly coupled, so we're trying to build an end to end testing framework. So from my perspective I'm living in a world where our end-to-end test suite doesn't exist and therefore could be equivalent to "killing it" and it is bad. Each component tests its own contracts, but if there's no…

sorry for the bait. thanks for the click :-D

Re: We killed our end-to-end test suite

#202

Post by alex-freire » Sat, Sep 25, 2021, 1:21 AM UTC

exdsq wrote: post #23

Ahhh I've made this mistake before. You can't test every input/output while also losing the ability to accurately depict stateful user flows. Even Fintech projects I've worked on at the >£10Bn daily volume mark combining formal methods, mathematical proofs, property based tests, fuzzing, model based testing, etc.. still caught issues pre-production via good old end-to-end tests. Good luck nubank :)

thank you

Re: We killed our end-to-end test suite

#203

Post by strken » Sat, Sep 25, 2021, 1:22 AM UTC

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

I've been thinking about the benefits of only writing E2E smoke tests which cover a small number of critical paths quickly. Seems like most of their problems came about because they wrote more tests than they needed, with higher coverage than was necessary.

Re: We killed our end-to-end test suite

#204

Post by alex-freire » Sat, Sep 25, 2021, 1:22 AM UTC

mjul wrote: post #51

In summary they noticed that their e2e suite mostly caught integration errors where clients and servers had incompatible schemas for the data exchange. The novelty is that they found a much faster way to identify this kind of errors by collecting and comparing the client and server side schemas statically without even running the code. This is a great optimisation, but it did not remove all defects so they still need…

well put! thanks for the great summary.

Re: We killed our end-to-end test suite

#205

Post by eyelidlessness » Sat, Sep 25, 2021, 1:28 AM UTC

CobrastanJorji wrote: post #123

eyelidlessness wrote: post #121

I’m generally not fond of e2e or even integration testing. At least, I prefer to keep them to a minimum, and use other tools to ensure units interact as expected. That said, where e2e tests may be valuable but costly as described in the article, it occurs to me that narrower integration tests which invert responsibility may be better. Which is to say: - Given Service A - Given Service B which depends on Service A Int…

What are the tools you do use to ensure units interact as expected?

Mostly making types more specific. Within a service, ensuring semantics are part of the type so they can be checked statically, and designing function boundaries so those semantics are part of the interface.

Between services, using standardized machine-usable docs (like OpenAPI and JSONSchema) to share those types in a well-defined way. This is harder because network boundaries are less flexible, but it does help a lot.

Re: We killed our end-to-end test suite

#206

Post by alex-freire » Sat, Sep 25, 2021, 1:29 AM UTC

AmpsterMan wrote: post #191

joshribakoff wrote: post #189

I’m not sure I follow the logic that their e2e test suite would take an “infinite” amount of time to run by 2021. It seems like an obviously faulty calculation, unless someone puts an infinite loop.

I think what they meant is that at the pace they were committing coffee to production, the e2e suite would never stop running. This is the most charitable interpretation.

you’re right sir!

Re: We killed our end-to-end test suite

#207

Post by physicles » Sat, Sep 25, 2021, 1:44 AM UTC

There’s the classic diagram showing tests as a pyramid[1]. At the bottom you have unit tests, white box stuff that mocks all dependencies and runs super fast. At the top you have E2E tests (or acceptance tests or whatever, names are fungible), but very few of them, to catch bugs at the boundaries between systems. In the middle you have stuff that maybe runs against a live database instead of a mock.

As you go further up the pyramid, tests get more expensive in every dimension: they are slower (and are run less frequently as a result), more expensive to debug and maintain, and maybe a bit flaky, though you should still de-flakify these tests as much as is practical.

The key is to push tests as far down the pyramid as possible. Never test something in an E2E test if it can be meaningfully tested in a unit test. For example, a regression test for a database date/time serialization bug should run against a real database. Meanwhile, a test verifying that an HTTP service returns the correct response code in a specific situation can run against mocks.

1 https://martinfowler.com/articles/practical-test-pyramid.htm... (holy crap that is a mountain of text, but the diagram is near the top)

Re: We killed our end-to-end test suite

#208

Post by BurningFrog » Sat, Sep 25, 2021, 1:58 AM UTC

jrochkind1 wrote: post #153

amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have. What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?

Not OP, but testing groups of the "units" you unit tests can have a lot of value.

Re: We killed our end-to-end test suite

#209

Post by AndrewKemendo » Sat, Sep 25, 2021, 2:03 AM UTC

This sounds like a garden variety example of an organization moving to event-based architecture and implementing contract testing.

What makes this interesting is that they had decided early on to be a Clojure/Lisp shop so there weren't great tools out there at enterprise scale.

I don't understand why they decided to built everything on Lisp, I assume it was best for them at the beginning. But given the known lack of tools for Lisp to scale into large scale enterprises - this is the kind of thing I would always consider once you get past MVP phase. Otherwise you have to end up building all your own stuff.

Architecture matters.

Re: We killed our end-to-end test suite

#210

Post by physicles » Sat, Sep 25, 2021, 2:10 AM UTC

gorgoiler wrote: post #179

The sad fact of E2E is that the tests genuinely find broken stuff. The “false negative” test results usually just mean false as in “something was broken, just not what the CI claimed was broken.” It’s could be anything , so you need automatic specificity as to what’s broken (hard) or buy-in from the entire organisation to be on standby for finding broken stuff (also hard.) “Anything” as in if your external DNS provid…

This is our goal with E2E tests and monitoring — write tests that fail if and only if customers are experiencing issues — and I don’t know how to achieve that level of I-can-sleep-well-at-night assurance of quality without it. We run our E2E tests continuously against prod from a different cloud region.

As you wrote, the challenge is that there are literally 100 things that could cause a test to fail (some of which are outside your control), and as your team scales you’ll have to get smart about how to efficiently dispatch people to fix problems.