Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headless Firefox, because PhantomJS is very bad software.
How to make Selenium tests reliable, scalable, and maintainable
11–20 of 71 posts
Re: How to make Selenium tests reliable, scalable, and maintainable
#12Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…
I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and out of the way when you're using it, but apparently is more true-to-reality than headless.
The test suite they wrote has about ~600 tests and while they're slower than I'd like (2~3 minutes) they've been bulletproof since we got my dev environment configured properly. It includes some fairly complicated interactions, most relevantly around our calendar interface.
Re: How to make Selenium tests reliable, scalable, and maintainable
#13The most annoying thing I found with Selenium was that it wouldn't wait for the browser to respond to click events and rerender. The approach in the blog post (and I think elsewhere ... not sure) is to poll the DOM with a timeout. Is there a better solution to be add with something like `executeScript`? You could run `requestAnimationFrame`, and then poll for an indicator that the click, etc. handler has indeed finis…
There's some utility methods like FluentWait, but ultimately they're just convenience wrappers for polling the DOM and waiting.
Re: How to make Selenium tests reliable, scalable, and maintainable
#14Also, by switching from Java to Ruby ecosystem is one way to improve your selenium tests. For start, use watir-webdriver and page-object gems.
Re: How to make Selenium tests reliable, scalable, and maintainable
#15The most annoying thing I found with Selenium was that it wouldn't wait for the browser to respond to click events and rerender. The approach in the blog post (and I think elsewhere ... not sure) is to poll the DOM with a timeout. Is there a better solution to be add with something like `executeScript`? You could run `requestAnimationFrame`, and then poll for an indicator that the click, etc. handler has indeed finis…
I have had some good results using the F# canopy library(http://lefthandedgoat.github.io/canopy/) for working with selenium. It handles (most) all the waits for you so you don't have to scatter a bunch of sleeps in your tests and is pretty easy to work with.
Re: How to make Selenium tests reliable, scalable, and maintainable
#16Some very good information in this article.
It is true that Selenium has its quirks, retrying a failed test can sometimes result in a passing test.
Disclaimer: I work for https://testingbot.com : at my work we offer our customers automatic retries when a test fails. Writing a Selenium test does take its time, but once you run it in parallel across hundreds of browser and os combinations, it's worth it.
Re: How to make Selenium tests reliable, scalable, and maintainable
#17Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…
This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable.
We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...
Re: How to make Selenium tests reliable, scalable, and maintainable
#18Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…
I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and…
I have trouble with Selenium's failure rate too. End up writing a test engine in Javascript. It handles async js func call with js function callback when finished to get rid of all the sleep-wait-retry type logic in Selenium.
Works very well. I can run the 100+ test cases in all IE/FF/Chrome/Safari, ios/android browser without change one line of JS/Test code. Runs fine with desktop with wire connect to cellphone browser on cell connection.
It tests out all the app backend db logic also. The time/pass/fail info are submitted back to the test backend db.
Re: How to make Selenium tests reliable, scalable, and maintainable
#19Also, by switching from Java to Ruby ecosystem is one way to improve your selenium tests. For start, use watir-webdriver and page-object gems.
Using Capybara alone one gets most of the stuff they had to implement (page, with, retries, ...) but I'll look into those gems you suggest.
Maybe the Scala ecosystem is still immature on the side of integration testing. They could implement them in Ruby if they are familiar with the language. I don't feel OK about using two languages but at least it could enforce strict separation between integration testing and the application.
Re: How to make Selenium tests reliable, scalable, and maintainable
#20The most annoying thing I found with Selenium was that it wouldn't wait for the browser to respond to click events and rerender. The approach in the blog post (and I think elsewhere ... not sure) is to poll the DOM with a timeout. Is there a better solution to be add with something like `executeScript`? You could run `requestAnimationFrame`, and then poll for an indicator that the click, etc. handler has indeed finis…
Ruby's Capybara encapsulates Selenium and waits until elements appear on the page (the default timeout is 2 seconds). So you can write simple sequential code like
click_link('bar')
expect(page).to have_content('baz')
Documentation at https://github.com/jnicklas/capybara#asynchronous-javascript...