100,000 e2e selenium tests? Sounds like a nightmare
watirmelon.com
100,000 e2e selenium tests? Sounds like a nightmare
1–10 of 58 posts
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#2> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week.
Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite.
I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great deal of user input they wind up only being a CRUD layer test.
I've gone the salesforce route, instead of using selenium all my "unit tests" are executed against the rest api.
In order to test "Deleting a product from the system" I have to create a user, assign the user as an administrator, log off of the root admin, log in as the new user, create a product, delete a product, get the product list and verify the product isn't on the list.
Typically this would be answered in a unit test like this:
p = productDao.create("My New Product")
assert true p.delete()
ymmv
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#3Why do we have to choose one or the other? Its ridiculous to stack these test methodologies against each other. You need both for different reasons. Unit tests verify the function of a unit. System-level tests verify the integration of units with other units. Instead of writing an exhaustive list of both, you learn over time the kinds of tests you need for each scenario.
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#4> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…
No-one ever said that integration tests and unit tests are mutually exclusive.
Integration test the happy paths, small integration suite that proves everything works together.
Unit test the individual units of functionality or behavior, whichever you're more comfortable with. Unit tests aren't supposed to tell you that the whole system works. Unit tests exist for two main reasons: Confidence that you can refactor your code without breaking functionality, and proving that your components are doing what they're supposed to do. Yes you technically can accomplish this with just an integration test suite but you will find refactoring bugs much harder to track down because the test failures aren't telling you exactly what went wrong.
You can never have full confidence in a system with just one suite of tests. You really need multiple suites. And when you TDD it all the way down (Integration test first, then work through the stack TDD-ing each part, when done Integration test and all unit tests should pass, and feature is done!) it's not a chore but actually quite enjoyable.
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#5> Now if we could just convince SalesForce to be more like Airbus and not fly a complete plane (or 50,000 planes) to test everything every-time they make a change...
I agree with the overall thrust of the article. Unit testing is indispensable, especially for me as a rubyist where there is basically nothing checked before runtime. But I find this conclusion that SalesForce is "doing it wrong" to be just as arrogant and presumptuous as DHH's blustery approach.
Now I don't know anything about SalesForce's architecture, but one thing I do know is that no system of that size exhibits the same ideals that can be maintained by a small team of brilliant developers working on the bleeding edge of some greenfield project with a set of stakeholders that fits in a single room. That's not to say that SalesForce couldn't use unit tests, but just that it's the height of arrogance for us as outsiders to presume one way or the other.
Scaling software development isn't solved by strict adherence to the dogma of idealistic methodologies any more than scaling traffic is solved by plugging in a "web-scale" nosql database into your Heroku. The reality is that scaling is done by removing bottlenecks over and over until the system that remains is an organic result reached through evolution rather than any architect's brilliant grand design.
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#6I find it strange that anyone would opt to move entirely away from one type of testing at all.
The benefits between unit tests and e2e system tests are varied and surely that's the main benefit?
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#7Why do we have to choose one or the other? Its ridiculous to stack these test methodologies against each other. You need both for different reasons. Unit tests verify the function of a unit. System-level tests verify the integration of units with other units. Instead of writing an exhaustive list of both, you learn over time the kinds of tests you need for each scenario.
I tend to agree - but if I could redraw that triangle it would be more like a trapezium. Not enough emphasis is given to system tests - and it shows in the tooling and the community mentality.
On my current project no one would dream of adding a new feature without unit tests - but everyone turns a blind eye to integration and system tests because "they're hard".
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#8> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…
I think it depends how tightly your business logic and your view are bound and how well your business logic is abstracted. I use Angular at work and 90% of the bugs we get can be caught by a unit test. On a rare occasion something will be mis-labeled on the markup and the two-way-binding "fails" silently, but usually our bugs are errors in the logic.
Whenever you fix a bug in your code, try to cover it with a unit test. If you can't, ask yourself whether it might be because the code is poorly abstracted.
I think there are definitely varying degrees of value in unit tests. Your example is definitely not super valuable, but unit tests on higher level methods can save a lot of trouble. When I code I want to rest assured that the different pieces I am using work. I want assurance that this class I'm about to use has an API that does exactly what it says. If I want to change modify the class, I want to see exactly what assumptions I'm breaking that users of this class might be making, and I want to be able to do this in under a second. This can have a drastic impact on developer velocity. I don't know of any other way of doing this than unit tests.
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#9> I would definitely choose a comprehensive suite of automated unit tests over a comprehensive suite of end-to-end/system tests any day of the week. Huh, after working on very large applications with tens of thousands of unit tests I'd choose the opposite. I've found little value from unit tests. They're sometimes good enough for catching the lowest common errors. With very complex applications that involve a great d…
Unit test are more useful for refactoring, making sure the system works the same before & after.
Re: 100,000 e2e selenium tests? Sounds like a nightmare
#10Why do we have to choose one or the other? Its ridiculous to stack these test methodologies against each other. You need both for different reasons. Unit tests verify the function of a unit. System-level tests verify the integration of units with other units. Instead of writing an exhaustive list of both, you learn over time the kinds of tests you need for each scenario.
That's the point of the testing pyramid at the end of the article. My rule of thumb is this: if you can test something at the unit testing level with mocks and feel confident about the behavior then do so. Otherwise, start moving up the pyramid (isolated integration tests > end of end integration tests > UI tests).
The higher you get the more time it takes for tests to run, the more difficult it is to identify where a failure is, the more brittle those tests will be because of inter-dependencies, and the more complex it is to write the test. The difficult part is understanding your system well enough to know where your unit tests aren't sufficient.