We killed our end-to-end test suite • Hacker News Board

Re: We killed our end-to-end test suite

#241

Post by amw-zero » Sat, Sep 25, 2021, 5:11 PM UTC

jrochkind1 wrote: post #153

amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have. What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?

Picture a test that doesn’t involve any clicking of UI elements. That’s a start. So right there you’re avoiding the complexities of UI rendering, you just call the commands that are invoked by clicking directly.

Also, you can test frontend components together, and backend components together, but not cross the client-server boundary. Faster, more reliable.

That leaves a very small amount of e2e tests that you even want to write, and by that point I’m totally fine with manual smoke testing or automating them. But they’re the vast minority of tests.

Re: We killed our end-to-end test suite

#242

Post by amw-zero » Sat, Sep 25, 2021, 5:12 PM UTC

mbrodersen wrote: post #216

amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

I flat out don’t believe you.

Re: We killed our end-to-end test suite

#243

Post by amw-zero » Sat, Sep 25, 2021, 5:15 PM UTC

sumedh wrote: post #235

amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

> At least for web applications, all end to end test suites are slow and flaky. Then you should be asking why are they flaky in the test environment? Its probably because your services are running on very slow servers.

No, it’s because end to end tests are non-deterministic with respect to execution time. They are deterministic only in their discrete states.

This is why all end to end testing involves polling to wait for asynchronous operations to complete, which is by definition non-deterministic.

Re: We killed our end-to-end test suite

#244

Post by amw-zero » Sat, Sep 25, 2021, 5:16 PM UTC

dcow wrote: post #157

amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

Sounds like you’ve been subject to some pretty poor test setups. I’ve experienced good ones. My cynical take is that well maintained e2e tests aren’t a product priority in environments where they’re flaky and slow so they come as an afterthought. Not that they can’t be good. Usually product wants to ship code yesterday and doesn't care if there are bugs… so good test hygiene is nowhere to be seen.

This argument does not account for the fact that all e2e tests are non-deterministic, so the quality of your “setup” is not relevant.

Re: We killed our end-to-end test suite

#245

Post by amw-zero » Sat, Sep 25, 2021, 5:18 PM UTC

mattnewton wrote: post #149

amw-zero wrote: post #144

Earlier quoted context omitted.

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I should be able to test that this usually works though, right?

This is what’s known as “counterintuition.” You would think that you could, but you are wrong.

I’m not saying you can’t write a passing end to end test. Of course you can get it to pass some times. But they are inherently non-deterministic.

Re: We killed our end-to-end test suite

#246

Post by amw-zero » Sat, Sep 25, 2021, 6:55 PM UTC

aidenn0 wrote: post #160

amw-zero wrote: post #144

Earlier quoted context omitted.

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I mean the exact output given a certain set of inputs may be slightly different due to asynchrony, but given a set of inputs, there should be a finite set of correct outputs and check for those. To use a stupid example: if listAnimals returns [cat, dog, mouse] some of the time and [cat, mouse, dog] other times, if your passes on the former and not the latter, then your test is broken and you should fix it. If it some…

A more accurate way to look at this based on your example is that, sometimes listAnimals returns [cat, dog, mouse], and sometimes it returns null.

It’s not that the result is nondeterministic, it’s that _whether or not the result is returned within the timeout of the polling mechanism_ is deterministic.

Re: We killed our end-to-end test suite

#247

Post by amw-zero » Sat, Sep 25, 2021, 6:57 PM UTC

aidenn0 wrote: post #175

grumple wrote: post #156

Earlier quoted context omitted.

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are pr…

If you use other people's software and hardware, and those things don't perform the way your software assumes they perform, knowing that would be useful, right? There's always a limit to how much you want to handle, but if you are having a test fail even a large fraction of 1% of the time, then there's probably some underlying behavior that you should account for in production as well.

No, that test doesn’t give you any useful information, because all it told you was that your expected answer wasn’t found in the configured time interval. You have no way of knowing whether or not your expected behavior would be satisfied if you ran for t + 1 seconds.

Re: We killed our end-to-end test suite

#248

Post by joshribakoff » Sat, Sep 25, 2021, 7:32 PM UTC

AmpsterMan wrote: post #191

joshribakoff wrote: post #189

I’m not sure I follow the logic that their e2e test suite would take an “infinite” amount of time to run by 2021. It seems like an obviously faulty calculation, unless someone puts an infinite loop.

I think what they meant is that at the pace they were committing coffee to production, the e2e suite would never stop running. This is the most charitable interpretation.

The way we solve that at Brex it scales logarithmically, we use bors merge bot which batches everyone’s PRs together and will binary search for the offending PR if any build fails. I’ve hardly ever waited longer than 2x the length our suite takes. They could also decouple CI and only run code tests for upstream changes, so it wasn’t obvious what they meant, thanks for clarifying one theory. I’m not sure I’d agree that it’s a problem with e2e tests per se but rather the problem is slowness itself, and monolithic CI. Not doing e2e tests can be a valid tradeoff to avoid slowness but I’d also point out these issues can be solved without deleting the e2e suite…

Re: We killed our end-to-end test suite

#249

Post by Veuxdo » Sat, Sep 25, 2021, 7:50 PM UTC

mbrodersen wrote: post #218

Veuxdo wrote: post #56

Earlier quoted context omitted.

E2E tests aren't worth it if they produce false positives and don't prevent defects from reaching production. By definition. Too many devs treat automated testing as a goal in and of itself.

So you are saying that X is not worth it if X is crap? Who wouldn’t agree with that? What if X was implemented well and did what it was supposed to do? We use end-to-end production and it works extremely well. I have had zero production problems for years now. All issues were found by the end-to-end tests before deploying.

"So you are saying that X is not worth it if X is crap?"

I am because it needs to be heard. Like I said, too many devs treat writing tests as a goal unto itself.

Re: We killed our end-to-end test suite

#250

Post by jerome-jh » Sat, Sep 25, 2021, 8:56 PM UTC

amw-zero wrote: post #247

aidenn0 wrote: post #175

Earlier quoted context omitted.

If you use other people's software and hardware, and those things don't perform the way your software assumes they perform, knowing that would be useful, right? There's always a limit to how much you want to handle, but if you are having a test fail even a large fraction of 1% of the time, then there's probably some underlying behavior that you should account for in production as well.

No, that test doesn’t give you any useful information, because all it told you was that your expected answer wasn’t found in the configured time interval. You have no way of knowing whether or not your expected behavior would be satisfied if you ran for t + 1 seconds.

After some time you have to consider the test is failed and investigate, even if it would have succeeded had the timeout been 1 second larger. I cannot believe they do not have quality of service requirements. Testing those requirements is of course not easy. It may take to much time to run on every release or may be considered out of the scope of E2E tests and compliance is checked with telemetry results.

However pick any response time mandated by the QOS requirements, multiply by an appropriate x and use this as the pass/fail timeout for your test. Take a value large enough that can easily be considered a bug (because e.g. the customer would think the operation failed and would hit refresh or back). You then have an issue that is definitely worth investigating. You may actually have reproduced a rare issue that is part of the long tail of your telemetry.