Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make Selenium tests reliable, scalable, and maintainable

lucidchart.com

Post reply on HN
Print view View original

31–40 of 71 posts

Re: How to make Selenium tests reliable, scalable, and maintainable

#31

Post by crdoconnor »

paulddraper wrote: post #4

The most annoying thing I found with Selenium was that it wouldn't wait for the browser to respond to click events and rerender. The approach in the blog post (and I think elsewhere ... not sure) is to poll the DOM with a timeout. Is there a better solution to be add with something like `executeScript`? You could run `requestAnimationFrame`, and then poll for an indicator that the click, etc. handler has indeed finis…

>Is there a better solution

Yes. And it's pretty simple:

    WebDriver driver = new FirefoxDriver();
    driver.get("http://somedomain/url_that_delays_loading");
    WebElement myDynamicElement = (new WebDriverWait(driver, 10))
  .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));
From : http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp

Re: How to make Selenium tests reliable, scalable, and maintainable

#32

Post by crdoconnor »

jmickey wrote: post #17
stevebmark wrote: post #11

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

>the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable.

What was actually going wrong during that 10%?

I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this.

Do you make heavy use of sleeps?

Re: How to make Selenium tests reliable, scalable, and maintainable

#33

Post by crdoconnor »

defied wrote: post #16

Some very good information in this article. It is true that Selenium has its quirks, retrying a failed test can sometimes result in a passing test. Disclaimer: I work for https://testingbot.com : at my work we offer our customers automatic retries when a test fails. Writing a Selenium test does take its time, but once you run it in parallel across hundreds of browser and os combinations, it's worth it.

>retrying a failed test can sometimes result in a passing test.

This is usually a sign of either a buggy test or buggy code.

Re: How to make Selenium tests reliable, scalable, and maintainable

#35

Post by jmickey »

crdoconnor wrote: post #32
jmickey wrote: post #17

Earlier quoted context omitted.

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

>the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. What was actually going wrong during that 10%? I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this. Do you make heavy use of sleeps?

Mostly these would be cases where the browser would, seemingly at random, end up in an unpredictable state and all proceeding test scenarios would fail because of this. (Page is white, or a completely unrelated website gets opened. We have seen lots of weird situations so far)

This might be exacerbated by the fact that we use the remote Browserstack Selenium hosting service so that the tests can be executed automatically as a part of our deployment process.

Re: How to make Selenium tests reliable, scalable, and maintainable

#36

Post by tommyd »

novocaine wrote: post #23

Everyone in the blogosphere (and at my own company) writing non-app-specific layers on top of selenium suggests that there is scope for a higher level framework that can be used on top of selenium. Or that the selenium api is too thin a layer over webdriver. Does anyone know of such a project?

You might find Site Prism interesting: https://github.com/natritmeyer/site_prism (there are alternatives such as http://watirwebdriver.com/page-objects/ and https://github.com/sensiolabs/BehatPageObjectExtension, but I have no experience with them).

It provides a "page object model" implementation on top of Capybara, so you can define a model for each page you want to test, which stores the page's relative URL, and has references to all the elements on the page you care about, and methods for all the interactions you want to do with that page.

So for example, you might have a "LoginPage" model, which contains the following:

  class LoginPage
Then whenever you want to login from one of your steps, you can just do:

  login_page = LoginPage.new
  login_page.login('whoever', 'what3v3r')
I think it's a nice abstraction as it allows more experienced test automation developers to build the page model while less experienced ones can write steps just calling the methods. You still have to pay a lot of attention to things like appropriate use of "wait for element to appear" rather than "sleep", and ensuring tests use isolated data, to get it working reliably, but we've got it working pretty well at my current place.

I should write up how we have it set up at some point as we have our own app-specific framework on top of SitePrism which provides some useful abstractions to make it quicker to develop tests.

Re: How to make Selenium tests reliable, scalable, and maintainable

#37

Post by bobm_kite9 »

The PageObjects tip is a really good one. Previously using Selenium you end up with a complete maintainability nightmare.

I used Geb on a recent project, and I actually felt that the tests I built demonstrated a passable level of engineering discipline. However, Geb was really hard to learn (partly the error messages were really confusing/missing) and you're still on top of Selenium so you still get wacky exceptions and edge cases.

Re: How to make Selenium tests reliable, scalable, and maintainable

#38

Post by masklinn »

rtpg wrote: post #24
stevebmark wrote: post #11

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…

Does it really? Does it test for whether a button you thought was present isn't actually clickable? If you're going to write tests, I think it makes an insane amount of sense to emulate real world conditions as much as feasibly possible (making judgement calls on things that don't matter like speed of the mouse).

Most Selenium tests don't test that a button is actually clickable though, they find things through the DOM and if the button is offscreen or hidden they won't realise it.

Re: How to make Selenium tests reliable, scalable, and maintainable

#39

Post by boundlessdreamz »

logn wrote: post #27
patio11 wrote: post #12

Earlier quoted context omitted.

I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and…

I had been using Firefox driver in Xvfb but wasn't happy with the performance/stability. So I built a Selenium driver out of Java only (using JavaFX's embedded WebKit) and used a headless JRE windowing toolkit (Monocle). My project is still a pre-release but the headless capability, Java-only system requirement, and its ajax handling might make it useful to some people currently: https://github.com/MachinePublishers/…

This looks very interesting!

1. How does it compare with phantomJS?

2. What's the current webkit version?

3. How often does the javaFX webkit update?

Re: How to make Selenium tests reliable, scalable, and maintainable

#40

Post by boundlessdreamz »

tkinom wrote: post #18
patio11 wrote: post #12

Earlier quoted context omitted.

I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and…

I have trouble with Selenium's failure rate too. End up writing a test engine in Javascript. It handles async js func call with js function callback when finished to get rid of all the sleep-wait-retry type logic in Selenium. Works very well. I can run the 100+ test cases in all IE/FF/Chrome/Safari, ios/android browser without change one line of JS/Test code. Runs fine with desktop with wire connect to cellphone brow…

Are you planning to open source it?

Can you give an example of how it works? Say navigate to a page, fill in a form, click submit and verify that some text is present after submitting the form

Post reply on HN