101–110 of 270 posts
End-to-end test suites do not excel in continuous integration workflows. They excel as part of your metrics, monitoring and alerting system running continuously for as long as the service they exercise lives. Perhaps it wouldn't be as painful if they took this approach instead.
This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…
> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in additio…
You really need to stop repeating this, it's absurd and there was a great post here the other day explaining how in most companies, they had a large QA team that would need to approve any code, it's just that developers were not expected to write the tests themselves.
> If the costs of writing and maintaining E2E tests outweigh the benefits, then obviously it's not worth it.
Obviously, but the question is, what's the alternative? In the blog post, the alternative was to have contract-based acceptance tests... but that may not always be appropriate for every business. We have a huge E2E test suite where I work and I was one of the biggest contributors to creating it... as everywhere else, it's heavy, slow and hard to maintain, but replacing what we have with contract testing wouold be unfeasible because we're not a micro-service architecture, we are one big application as we're a product company.... I would love to find a better way of testing our product, but contract-based testing is definitely not the answer for us.
Earlier quoted context omitted.
I always mocked out 3rd party tests in my tests. I've never actually had a problem with some third party changing their API. That's the whole point of a versioned API anyway. I think when people talk about e2e tests, it's more about testing only integration between contracts that you own.
I’ve run into this a few times with some upstream package breaking and showing up in tests. I try to avoid mocking as much as possible in tests these days.
That said, I too avoid mocks unless there's a specific good reason to add one.
This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…
They didn't say E2E aren't worth it (they actually said it worked for them early on) Their point is that it's not scalable as they think contract testing is
Personally, I think they are wrong, and the problems they have with their E2E tests are problems with their implementation. Fixing them would benefit the customer experience and the developer experience as well as reducing the costs of the E2E. They are absolutely going to have critical customer impact (they are fintech ffs) that their E2E tests would have caught. Of course, whether that actually tanks their business depends on other factors. So accepting more critical issues may the right thing for the success of the company. Robinhood customers were greatly pissed off by its behavior, and yet it doesn't seem to have hurt them too much. But I wouldn't fucking boast about it!
Earlier quoted context omitted.
Types are compile checks, they have nothing to do with contracts, having contracts for messages that will transit over a queue are still useful even if you are using a typed language.
Not in all environments, e.g. clojure.spec, or with MyPy reflection etc.
For example, imagine you have two services that communicates through a message queue. Service A produces X as a string, but Service B consumes X as an integer. You can type that, both services would compile, but it would break as soon as you tried to consume that message. And yes, you can build something using MyPy reflection or whatever, but you have to build it anyway.
Earlier quoted context omitted.
Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests. They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e…
> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments. ...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties? (I know my answer, but I'm curious to hear yours.)
Of course, both have the same form, you run the system and verifies if the results match the expected. But monitoring is done constantly during the lifetime of your infrastructure, and verifies the entire infrastructure; while tests are done episodically, and verifies your program or a component. Tests also often block some procedures, while monitoring doesn't (but it certainly starts some).
The middle ground that not enough teams are exploring is following the so-called Functional Architecture. If all side-effects are effectively segregated and reified, then one should be able to swap them out for determistic mocks that run instantly. So you could E2E a distributed system realistically and instantly. You can still simulate things services being slow, unavailable, etc if the code handling those is expres…
exactly! well said. it's what we're trying to achieve with our acceptance testing strategy. you can see more here how we leveraged clojure to be able to simulate E2E in memory in the JVM by bypassing IO and just having one services logic+data layer talk to the other: https://www.youtube.com/playlist?list=PLfqo9_UMdHhah_gNPnawX...
Looking forward to eventually check out Sachem if there's a plan to share it?
Earlier quoted context omitted.
And in part 2 about 12 minutes in we show the code to exemplify how our acceptance test suite works (and is very different from traditional E2E): https://www.youtube.com/watch?v=caxpxszueI0&list=PLfqo9_UMdH...
I cannot watch this entire video, so it is possible that you explain conceptually how your acceptance tests are different from end to end test? I would like to undestand what were the goals of your initial end to end and what are the goals of acceptance testing, and how you define these acceptance testing from test objective perspective. I assume end to end tests could be described by this definition: "test the funct…
FWIW I've been in both situations. One company had sketchy E2E coverage that resulted in a modicum of production bugs. I moved to a competitor of roughly same size had a huge E2E suite and AFAICT results in roughly the same modicum of production bugs. But feature development at the latter moves much more slowly because of all the wait queues, flaky tests, timeouts, test maintenance overhead, etc. IMO these seem to be…
1. No e2e tests.
2. Heavy, flaky, slow e2e tests.
3. e2e as a driver of first class system integration
1. no unit tests
2. tons of unit tests, that regularly fail, nobody cares and check in more bad tests, "unit" tests that thread sleep and take minutes
3. CI/CD with 0 tolerance for failures or >1s tests