We killed our end-to-end test suite
building.nubank.com.br
We killed our end-to-end test suite
1–10 of 270 posts
Re: We killed our end-to-end test suite
#2Re: We killed our end-to-end test suite
#3Re: We killed our end-to-end test suite
#4If you say so, and I wish you luck but... I've seen that tried many, many times and never seen it actually work out in practice. It seems like it ought to be workable - there are only a finite number of ways that each service can be invoked after all - but if the goal of automated testing is to find problems before they become production problems, I've never seen "defined contracts" fulfill that goal.
Re: We killed our end-to-end test suite
#5E2E tests are very hard to maintain but in many situations they are required.
Re: We killed our end-to-end test suite
#6They excel as part of your metrics, monitoring and alerting system running continuously for as long as the service they exercise lives.
Perhaps it wouldn't be as painful if they took this approach instead.
Re: We killed our end-to-end test suite
#7Re: We killed our end-to-end test suite
#8I love property-based testing, especially with these new frameworks now does coverage-guided fuzzing too. However, it only guarantees the "contract" (or "interface") at that level. For property-based testing (or contract testing this article calls) today, it is still very much to only validate the property holds true, not to exhaust all the edge cases. To give an example, a property-based testing validates function: "add(x, y) == add(y, x), given x in Int32 range", it doesn't validate edges cases what if you call "add" twice, 3 times, from different threads etc.
At the end of the day, it would be hard for the property-based testing to validate your component satisfy Liskov substitution principle.
Integration testing on the other hand, makes sure your system worked at integrated level. It doesn't enforce Liskov substitution principle too. However, if you have downstream components depend on your implementations (for example, there is an earlier callsite called a function before, second call must be cached), update the upstream and run integration testing makes that implicit assumption apparent.
So, that's where I am arriving at. Without a powerful language that can encode all contract at programming level, relying on property-based testing only at components level cannot maintain substitution invariant. Integration testing is required.
Re: We killed our end-to-end test suite
#9This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…
Or, from another perspective - you are doing end-to-end testing, the question is whether you're doing it before production or if your customers are doing it for you...