Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

211–220 of 270 posts

Re: We killed our end-to-end test suite

#211

Post by sidlls »

hinkley wrote: post #124
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

> The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests.

I'd view that as a win then. Unit tests are next to worthless for anything but tightly bound domains, like libraries, in most cases.

They're actively harmful in things like application-level service code, where they're used to turn perfectly well-written code into a chockablock mess to satisfy "testibility".

Re: We killed our end-to-end test suite

#212

Post by askonomm »

AndrewKemendo wrote: post #209

This sounds like a garden variety example of an organization moving to event-based architecture and implementing contract testing. What makes this interesting is that they had decided early on to be a Clojure/Lisp shop so there weren't great tools out there at enterprise scale. I don't understand why they decided to built everything on Lisp, I assume it was best for them at the beginning. But given the known lack of…

What enterprise tools are you thinking of? Because Clojure runs on the JVM, I'd imagine what works for Java also to work for Clojure.

Personally I see Clojure as a perfect choice for an ever-increasing amount of complexity, since you'd keep side-effects on the edges of your app and everything else as a pure function. Every occurrence of enterprise OO I have ever seen has been unmanageable mess that burns everyone out in a quick year, because you have no idea where something begins or ends, what goes on in-between, and instead of doing what actually matters (business logic) you spend most of your time creating abstractions.

Re: We killed our end-to-end test suite

#213

Post by mbrodersen »

The real problem is that they don’t know how to write fast end-to-end tests. Not having solid end-to-end tests is a massive red flag. Also, each service can be simulation tested in isolation. So even if they don’t have the skills to do end-to-end testing, they could at least do per-service simulation testing, with each service team responsible for running those tests independently.

Re: We killed our end-to-end test suite

#214

Post by exdsq »

alex-freire wrote: post #202
exdsq wrote: post #23

Ahhh I've made this mistake before. You can't test every input/output while also losing the ability to accurately depict stateful user flows. Even Fintech projects I've worked on at the >£10Bn daily volume mark combining formal methods, mathematical proofs, property based tests, fuzzing, model based testing, etc.. still caught issues pre-production via good old end-to-end tests. Good luck nubank :)

thank you

I'll be interested to see how it goes, will you follow up with a blog post in a couple months?

Re: We killed our end-to-end test suite

#215

Post by mbrodersen »

kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

Agree. Other organisations can effective run very large end-to-end tests. So it is most definitely possible. If they don’t have the in-house skills to do it then they should hire somebody who can.

Re: We killed our end-to-end test suite

#216

Post by mbrodersen »

amw-zero wrote: post #146
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There…

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

Re: We killed our end-to-end test suite

#217

Post by mbrodersen »

hinkley wrote: post #124
kodablah wrote: post #18

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

Unit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.

Re: We killed our end-to-end test suite

#218

Post by mbrodersen »

Veuxdo wrote: post #56
msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

E2E tests aren't worth it if they produce false positives and don't prevent defects from reaching production. By definition. Too many devs treat automated testing as a goal in and of itself.

So you are saying that X is not worth it if X is crap? Who wouldn’t agree with that? What if X was implemented well and did what it was supposed to do? We use end-to-end production and it works extremely well. I have had zero production problems for years now. All issues were found by the end-to-end tests before deploying.

Re: We killed our end-to-end test suite

#219

Post by R0b0t1 »

mbrodersen wrote: post #217
hinkley wrote: post #124

Earlier quoted context omitted.

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…

Unit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.

You're both kind of right. You need to E2E your product, and what your product is is what matters.

Re: We killed our end-to-end test suite

#220

Post by mbrodersen »

commandlinefan wrote: post #4

> Contract tests allow us to describe the interactions between our services through expectations to be met by their inputs and outputs. If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated te…

Agree. They are not testing the end-to-end contracts of how concurrent services interact. I predict it will end badly.
Post reply on HN