Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

251–260 of 270 posts

Re: We killed our end-to-end test suite

#251

Post by aidenn0 »

amw-zero wrote: post #246
aidenn0 wrote: post #160

Earlier quoted context omitted.

I mean the exact output given a certain set of inputs may be slightly different due to asynchrony, but given a set of inputs, there should be a finite set of correct outputs and check for those. To use a stupid example: if listAnimals returns [cat, dog, mouse] some of the time and [cat, mouse, dog] other times, if your passes on the former and not the latter, then your test is broken and you should fix it. If it some…

A more accurate way to look at this based on your example is that, sometimes listAnimals returns [cat, dog, mouse], and sometimes it returns null. It’s not that the result is nondeterministic, it’s that _whether or not the result is returned within the timeout of the polling mechanism_ is deterministic.

Presumably that happens in production as well, and the test can determine that the system does the proper thing when that happens?

Re: We killed our end-to-end test suite

#252

Post by aidenn0 »

jerome-jh wrote: post #250
amw-zero wrote: post #247

Earlier quoted context omitted.

No, that test doesn’t give you any useful information, because all it told you was that your expected answer wasn’t found in the configured time interval. You have no way of knowing whether or not your expected behavior would be satisfied if you ran for t + 1 seconds.

After some time you have to consider the test is failed and investigate, even if it would have succeeded had the timeout been 1 second larger. I cannot believe they do not have quality of service requirements. Testing those requirements is of course not easy. It may take to much time to run on every release or may be considered out of the scope of E2E tests and compliance is checked with telemetry results. However pi…

Right have a timeout measured in minutes. The timeouts have zero effect on a clean run, so large timeouts have no effect on time to deploy if you require a clean run of tests for deploying.

Re: We killed our end-to-end test suite

#253

Post by sumedh »

amw-zero wrote: post #243
sumedh wrote: post #235

Earlier quoted context omitted.

> At least for web applications, all end to end test suites are slow and flaky. Then you should be asking why are they flaky in the test environment? Its probably because your services are running on very slow servers.

No, it’s because end to end tests are non-deterministic with respect to execution time. They are deterministic only in their discrete states. This is why all end to end testing involves polling to wait for asynchronous operations to complete, which is by definition non-deterministic.

No not really, what you are saying is that you dont how long a test will take to complete, it can take 1 min or it can take 1 hour. If it sometimes takes 1 hour then you have to put on your detective hat and go look in the logs to see which service is slowing the e2e flow.

Re: We killed our end-to-end test suite

#254

Post by mbrodersen »

amw-zero wrote: post #242
mbrodersen wrote: post #216

Earlier quoted context omitted.

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

I flat out don’t believe you.

Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.

Re: We killed our end-to-end test suite

#255

Post by mbrodersen »

hinkley wrote: post #240
mbrodersen wrote: post #217

Earlier quoted context omitted.

Unit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.

You can’t test all error cases end to end. If you can you have shitty error handling. Clock skew between servers? Drifting clock skew? Disk space exhaustion? Disk space exhaustion at each possible failure point? There are so many of these and you’re going to inject most of them in unit tests. My original point was that if you can’t write good unit tests your e2e tests are also going to be lousy, and you will never ge…

Of course you can. Part of the end-to-end test is to setup the test scenarios you want to test (including limited HD space etc.)

Re: We killed our end-to-end test suite

#256

Post by mbrodersen »

hinkley wrote: post #240
mbrodersen wrote: post #217

Earlier quoted context omitted.

Unit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.

You can’t test all error cases end to end. If you can you have shitty error handling. Clock skew between servers? Drifting clock skew? Disk space exhaustion? Disk space exhaustion at each possible failure point? There are so many of these and you’re going to inject most of them in unit tests. My original point was that if you can’t write good unit tests your e2e tests are also going to be lousy, and you will never ge…

They are only slow if you do it wrong.

Re: We killed our end-to-end test suite

#257

Post by mbrodersen »

mbrodersen wrote: post #215
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

Agree. Other organisations can effective run very large end-to-end tests. So it is most definitely possible. If they don’t have the in-house skills to do it then they should hire somebody who can.

What’s incorrect about my comment? Clearly other organisations can write successful end-to-end tests. The fact that you can’t do it doesn’t mean that nobody can.

Re: We killed our end-to-end test suite

#258

Post by firedark60 »

mbrodersen wrote: post #216
amw-zero wrote: post #146

Earlier quoted context omitted.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

+1 on this comment. We had issues with poorly written selenium tests and after rewriting new tests with cypress and better test practices, the e2e test are reliable enough to be used as canary testing in new environments without false positive flaking. It ultimately comes down to how much you're willing to invest in writing good tests.

If you're suffering from seriously flakey e2e tests results, more often than not it's from either outdated tech, poor testing practices, and not enabling a selective retry on failures.

Re: We killed our end-to-end test suite

#259

Post by amw-zero »

mbrodersen wrote: post #254
amw-zero wrote: post #242

Earlier quoted context omitted.

I flat out don’t believe you.

Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.

My disbelief of you is from my own experiences writing them, and talking to dozens of colleagues across different companies.

Every single company has to heavily parallelize their e2e tests, and pays a huge CI bill on top of effort maintaining an overly complex CI config.

Even after this, every company has to have a retry mechanism for their e2e tests because at least one fails at least every test run. It’s also the first thing I ask on interviews. I have many dat points on this one, across many teams and companies.

It’s an abomination.

Maybe you’re not talking about interactive web applications. Or maybe you’re not talking about the scale where you have thousands of e2e tests written over multi-year projects. If you’re talking about a toy project, sure you might not run into nondeterminism. These problems present themselves in aggregate.

Re: We killed our end-to-end test suite

#260

Post by mbrodersen »

amw-zero wrote: post #259
mbrodersen wrote: post #254

Earlier quoted context omitted.

Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.

My disbelief of you is from my own experiences writing them, and talking to dozens of colleagues across different companies. Every single company has to heavily parallelize their e2e tests, and pays a huge CI bill on top of effort maintaining an overly complex CI config. Even after this, every company has to have a retry mechanism for their e2e tests because at least one fails at least every test run. It’s also the f…

I am talking about very large scale systems with web + mobile + desktop clients in an Enterprise environments. Think airlines and airports (operations and pairing/rostering). However it sounds as if your experience has been pretty bad. So I understand where you are coming from. My experience is that yes it is hard to do right but also that it is worth doing. YMMV of course.
Post reply on HN