Isn't this what protos are for?
We killed our end-to-end test suite
181–190 of 270 posts
Re: We killed our end-to-end test suite
#182Earlier quoted context omitted.
I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…
How do you run tests in parallel if part of the logic you are testing is a sql statement? Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.
In addition to the database pool approach, you can also write tests so that they are inherently independent. Each test creates and (optionally) deletes its own data, without making assumption about what else is in the database. That's not ideal, as it's hard to know you're not making a hidden assumption.
Re: We killed our end-to-end test suite
#183Earlier quoted context omitted.
You have been in two of three situations. 1. No e2e tests. 2. Heavy, flaky, slow e2e tests. 3. e2e as a driver of first class system integration Compare with 1. no unit tests 2. tons of unit tests, that regularly fail, nobody cares and check in more bad tests, "unit" tests that thread sleep and take minutes 3. CI/CD with 0 tolerance for failures or >1s tests The difference between "We have tons of tests" and "We driv…
>3. e2e as a driver of first class system integration Could you say more about this approach? That's how I've tried to approach end-to-end tests, but I haven't found much written about this which is specifically about e2e tests.
I haven't written a book but just take TDD philosophy but apply it to e2e. For example, if you have an e2e test that often fails because something times out, or you've had to set a high timeout, then dig down why and fix it. Turns out you have a JVM that often has a 2s GC but no alarms, and yes, it impacts customers too. Fix that problem, don't get rid of the test (or all the tests). You got a slow third party thing, put an abstraction in front of it so that username: MyE2EUser's traffic goes to a shim. Then either negotiate with third party, or make interaction with third party system asychronous. Or does resubmitting the page order the product twice?! lol. I wish those were the old days. All that being said, I haven't done this for a couple of years now, and I have the joys of vastly simpler systems at vastly huger scale, so I may be seeing the past through rose tinted glasses.
Re: We killed our end-to-end test suite
#184Earlier quoted context omitted.
I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc. I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are conside…
How do you run tests in parallel if part of the logic you are testing is a sql statement? Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.
One option I’ve used only works if there’s some natural partition of the data like a customer ID. Every test starts by creating a new customer account. Since by design customers can’t see each other’s data, therefore tests can’t interfere with each other and can run in parallel on a single database. After all, in production all your customers are going to be using the database at the same time right? So it needs to work anyway.
Re: We killed our end-to-end test suite
#185Earlier quoted context omitted.
> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra? We actually do pretty well testing against fake infra. We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightwe…
> At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. Yeah, you're not wrong. I'm just griping because of the previous list of complaints. There are some benefits to running your tests against a mirror infra to reality. But they are limited. The bugs they catch that you won't catch running fake infra are very small in number, but terrifying in diffi…
The article did make its points fairly poorly.
I think in the end they more or less did what you suggested as well they just didn't call it end to end testing.
Re: We killed our end-to-end test suite
#186Earlier quoted context omitted.
I can hardly think of a situation where I'd want no end to end test. I think one misconception is that there has to be a single end to end test. Really what you want is a variety of end to end tests examining the functionality of different parts of the system. But the system under test is still the whole system, not the units. These partial end to end tests can still be quick to run, as long as you keep system startu…
there was a nice talk putting this as an example where React's engineering teams designed their test to be future proof: always test the public API; your end to end tests should survive a major refactoring/rewrite
95% of our tests are going through the API, including all the auth. Spin up a server in the test, create some data (we have helpers), call the endpoint, assert on the response, another endpoint's response, or the database state.
That allowed us to carry out _enormous_ refactors (pretty much only the controllers stayed) without touching tests. It's not really harder to write, making an API request isn't harder to write than making a function call
Re: We killed our end-to-end test suite
#187Earlier quoted context omitted.
> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)
That's monitoring, not testing. Of course, both have the same form, you run the system and verifies if the results match the expected. But monitoring is done constantly during the lifetime of your infrastructure, and verifies the entire infrastructure; while tests are done episodically, and verifies your program or a component. Tests also often block some procedures, while monitoring doesn't (but it certainly starts…
You can try to monitor that an endpoint responds quickly but how do you monitor that it responds correctly? At the end of the day both tests and monitoring are forms of verification
Some people run (subsets) of their tests in production as a form of monitoring. Sometimes monitoring does not pass or fail and is instead qualitative like a dashboard or raw logging, without alerts
I’d say there is a grey area between monitoring and testing, it is more precise to ask if you’re verifying pre production, post production, or both
Re: We killed our end-to-end test suite
#188Earlier quoted context omitted.
> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)
I always mocked out 3rd party tests in my tests. I've never actually had a problem with some third party changing their API. That's the whole point of a versioned API anyway. I think when people talk about e2e tests, it's more about testing only integration between contracts that you own.
It’s valid to say you’re e2e testing your system, just not e2e testing the “full system”.
This is why the classification of the test into e2e, integration, and unit can cause confusion. I like to try to encourage people to avoid bucketing and instead say “this test should be more integration style than it currently is”, “this test should be more isolated than it currently is”. At the end of the day all testing mocks out the user and things like old web browsers or other factors that are a part of the real world system you care about may not be simulated in your test, so the way to get ”real” e2e verification is probably monitoring real users, if you consider that the user is a part of your “system”
Re: We killed our end-to-end test suite
#189I’m not sure I follow the logic that their e2e test suite would take an “infinite” amount of time to run by 2021. It seems like an obviously faulty calculation, unless someone puts an infinite loop.
Re: We killed our end-to-end test suite
#190Earlier quoted context omitted.
there was a nice talk putting this as an example where React's engineering teams designed their test to be future proof: always test the public API; your end to end tests should survive a major refactoring/rewrite
95% of our tests are going through the API, including all the auth. Spin up a server in the test, create some data (we have helpers), call the endpoint, assert on the response, another endpoint's response, or the database state. That allowed us to carry out _enormous_ refactors (pretty much only the controllers stayed) without touching tests. It's not really harder to write, making an API request isn't harder to writ…
Great stuff. Refactoring in my codebase is quite painful because earlier owners went in almost the opposite direction. The unit tests pretty much are never the ones that catch our bugs -- it's all caught by random or end to end tests. I'm slowly trying to crawl toward the light with respect to reorganizing tests I come across to use the external API.