191–200 of 270 posts
I’m not sure I follow the logic that their e2e test suite would take an “infinite” amount of time to run by 2021. It seems like an obviously faulty calculation, unless someone puts an infinite loop.
This is the most charitable interpretation.
This is an endless debate. Each situation requires a different test setup but ultimately you can't say end-to-end tests are not worth it. You can have perfectly functioning units of software that are all perfectly unit tested but the units are not working together (insert a related meme GIF about working drawers colliding when opened). This can happen with strongest inter-unit communication protocols such as strong t…
> you can't say end-to-end tests are not worth it You can, actually. But here's the thing: I've never seen an honest debate on E2E within an org. When your manager comes to you and says your team is going to start doing E2E, ask him/her if they are prepared for their schedule to slip by 30% or more. They will either slither back into their office, or (most likely) they will insist that developers write E2E in additio…
That’s just silly. Of course there were tests 15 years ago. Unit testing has been around since the 1950s
Very little would cause the world to end. But lives have been lost and billions of dollars along the way wasted due to improper testing
Earlier quoted context omitted.
The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…
While I agree that writing unit tests are a lot harder, and you develop good skills in attempting to write them, I must say that in the projects I worked on most bugs were caught by integration tests (technically not E2E tests), and not unit tests. I've also had projects with only unit tests, and almost no bugs were found by it, and there were plenty of bugs. Ideally, I would like both. But if I had to have only one,…
Eg, a b-tree has a bunch of invariants: Leaves have equal height, data is sorted, nodes have between N/2 and N values, they contain everything that was inserted and not deleted, etc. So write a test which makes random changes to a b-tree in a loop, and makes those same changes to a simple sorted list. Every iteration, verify the invariants hold and values match. Every 1000 iterations, throw out the object and start again with a new seed. If the test ever fails, print out the seed for easy reproducibility.
In your unit testing suite, run this fuzzer for about 100ms or something. This catches lots of bugs. And occasionally leave the randomiser running overnight looking for rare bugs.
This sort of thing is so humbling, for the sheer volume of “obvious” bugs you find in otherwise working code. It’s hands down the best value testing code I’ve ever written.
Queueing theory is an excellent way to look at E2E testing for the big-picture view and also for drilling down into each of the relevant services.
For a quick intro, this is a queueing theory primer that I've written and shared with HN previously: https://github.com/joelparkerhenderson/queueing-theory
Earlier quoted context omitted.
How do you run tests in parallel if part of the logic you are testing is a sql statement? Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.
One option I’ve used only works if there’s some natural partition of the data like a customer ID. Every test starts by creating a new customer account. Since by design customers can’t see each other’s data, therefore tests can’t interfere with each other and can run in parallel on a single database. After all, in production all your customers are going to be using the database at the same time right? So it needs to w…
Earlier quoted context omitted.
That's monitoring, not testing. Of course, both have the same form, you run the system and verifies if the results match the expected. But monitoring is done constantly during the lifetime of your infrastructure, and verifies the entire infrastructure; while tests are done episodically, and verifies your program or a component. Tests also often block some procedures, while monitoring doesn't (but it certainly starts…
You can try to monitor that an endpoint responds quickly but how do you monitor that it responds correctly? At the end of the day both tests and monitoring are forms of verification Some people run (subsets) of their tests in production as a form of monitoring. Sometimes monitoring does not pass or fail and is instead qualitative like a dashboard or raw logging, without alerts I’d say there is a grey area between mon…
Edit: That is to say, what distinguishes testing from monitoring isn’t content, but purpose.
Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue. Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds tr…
The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them. It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth. Often E2E tests…
Either alone is insufficient. Both together aren't necessarily sufficient.