Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We killed our end-to-end test suite

building.nubank.com.br

Post reply on HN
Print view View original

71–80 of 270 posts

Re: We killed our end-to-end test suite

#71

Post by rgoulter »

commandlinefan wrote: post #9
msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

> you can't say end-to-end tests are not worth it Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...

> you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you.

This reminds me of the point I've seen made elsewhere: E2E tests can be complemented by use of monitoring metrics, healthchecks, etc. for providing confidence that the system is working as intended (or for spotting cases where it's not working as intended).

Re: We killed our end-to-end test suite

#73

Post by X6S1x6Okd1st »

> One of our Sr Staff Engineers, Rafael Ferreira, ran some numbers and applied queueing theory.

Is it just me or is queueing theory seemingly misused all over the place in software engineering. I don't know what was applied here, as it's omitted, but I've certainly seen people suggest that doing things like mandating that cycle time is decreased will increase throughput and justifying what they are saying with "queue theory".

Mathematical constructs are great, but they aren't worth much if you can't ensure their constraints are met.

Re: We killed our end-to-end test suite

#74

Post by alex-freire »

kleinsch wrote: post #11

Buried ten feet deep in the article - they retired E2E tests and introduced "acceptance" tests, which are more efficient E2E tests that they still run on critical code. But I guess "We Renamed Our Eng-to-End Test Suite" isn't a very good blog post.

Thanks for the feedback, we didn't want to burry the "acceptance test" complement to our testing strategy in the article. If you're curious how it works we've recorded a webinar about it: https://www.youtube.com/watch?v=wKgDaD5Nie4&list=PLfqo9_UMdH...

Re: We killed our end-to-end test suite

#75

Post by munificent »

kodablah wrote: post #33
Daishiman wrote: post #26

Earlier quoted context omitted.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived". E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…

> is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.

Another way to say this is that efficient e2e tests require significant continuous investment in top-tier engineer time. The question then is how much engineering time is worth being spent in that way.

It may be that, yes, you can have fast e2e suites, but doing so is too expensive to justify the cost.

Re: We killed our end-to-end test suite

#76

Post by alex-freire »

alex-freire wrote: post #74
kleinsch wrote: post #11

Buried ten feet deep in the article - they retired E2E tests and introduced "acceptance" tests, which are more efficient E2E tests that they still run on critical code. But I guess "We Renamed Our Eng-to-End Test Suite" isn't a very good blog post.

Thanks for the feedback, we didn't want to burry the "acceptance test" complement to our testing strategy in the article. If you're curious how it works we've recorded a webinar about it: https://www.youtube.com/watch?v=wKgDaD5Nie4&list=PLfqo9_UMdH...

And in part 2 about 12 minutes in we show the code to exemplify how our acceptance test suite works (and is very different from traditional E2E): https://www.youtube.com/watch?v=caxpxszueI0&list=PLfqo9_UMdH...

Re: We killed our end-to-end test suite

#77

Post by chartpath »

fndex wrote: post #63
continuational wrote: post #34

> In our analysis, we figured out that the most frequent category of bugs caught by End-to-End tests was schema violations. Schema violations are pretty much just type errors. Fortunately these can be prevented automatically and with 100% confidence without writing even a single test.

Types are compile checks, they have nothing to do with contracts, having contracts for messages that will transit over a queue are still useful even if you are using a typed language.

Not in all environments, e.g. clojure.spec, or with MyPy reflection etc.

Re: We killed our end-to-end test suite

#78

Post by [deleted] »

deckard1 wrote: post #70
msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in additio…

[deleted]

Re: We killed our end-to-end test suite

#79

Post by alex-freire »

coryrc wrote: post #22
commandlinefan wrote: post #9

Earlier quoted context omitted.

> you can't say end-to-end tests are not worth it Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...

Tests are never 100%, customers will always find issues. They made the choice to allow slightly more bugs to production than before, not to go from never having bugs in production to having some.

we didn't see more bugs in production when we sunseted E2E. in fact, complementing contract tests with acceptance tests we saw less bugs in production and more productive test creation and maintanence. :-D

Re: We killed our end-to-end test suite

#80

Post by alex-freire »

Veuxdo wrote: post #56
msoad wrote: post #5

This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…

E2E tests aren't worth it if they produce false positives and don't prevent defects from reaching production. By definition. Too many devs treat automated testing as a goal in and of itself.

well said!
Post reply on HN