Live data from Hacker News

How to make Selenium tests reliable, scalable, and maintainable

lucidchart.com

51–60 of 71 posts

Re: How to make Selenium tests reliable, scalable, and maintainable

#51
post #5

Nice rundown, wish I had read this a year ago! > One developer designed a way to take a screenshot of our main drawing canvas and store it in Amazon’s S3 service. This was then integrated with a screenshot comparison tool to do image comparison tests. I would also take a look at Applitools https://applitools.com/ — they have Selenium webdriver-compatible libraries that do this screenshot taking/upload and offer a nic…

If using Selenium's Python bindings, you can take a screenshot from Selenium and convert it to OpenCV format like this:

    cv2.imdecode(
        numpy.asarray(
            bytearray(base64.decodestring(driver.get_screenshot_as_base64())),
            dtype=numpy.uint8),
        cv2.CV_LOAD_IMAGE_UNCHANGED)
(where `driver` is your WebDriver object, e.g. `WebDriver.Chrome()`).

Then to match that frame against a previously-captured "template" image, you can use stb-tester's[1] "match" function[2] which allows you to specify things like the region to ignore and tweak the matching sensitivity.

[1] http://stb-tester.com [2] http://stb-tester.com/stb-tester-one/rev2015.1/python-api#st...

Re: How to make Selenium tests reliable, scalable, and maintainable

#52

The most annoying thing I found with Selenium was that it wouldn't wait for the browser to respond to click events and rerender. The approach in the blog post (and I think elsewhere ... not sure) is to poll the DOM with a timeout. Is there a better solution to be add with something like `executeScript`? You could run `requestAnimationFrame`, and then poll for an indicator that the click, etc. handler has indeed finis…

>Is there a better solution Yes. And it's pretty simple: WebDriver driver = new FirefoxDriver(); driver.get("http://somedomain/url_that_delays_loading"); WebElement myDynamicElement = (new WebDriverWait(driver, 10)) .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement"))); From : http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp

I'm not sure this satisfies your parent poster's requirement of: "if it fails, you know about it pretty soon, without the need for long timeouts."

Re: How to make Selenium tests reliable, scalable, and maintainable

#53

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…

For testing Mithril.js, I wrote a mock window object, which allow you to do things like simulate requestAnimationFrame clicks, JSON-P calls and browser quirks from non-browser environments (e.g. from a Node.js script). So to test, you simply swap `window` with the mock and you can drive your fake browser however you wish.

http://lhorie.github.io/mithril/mithril.deps.html

You can cover a lot of ground with that approach and make an extremely fast test suite that is suitable for a save-refresh-test workflow and then you can put trickier tests in a secondary test suite that you only run once in a while (e.g. before a commit)

Re: How to make Selenium tests reliable, scalable, and maintainable

#54
post #20

The most annoying thing I found with Selenium was that it wouldn't wait for the browser to respond to click events and rerender. The approach in the blog post (and I think elsewhere ... not sure) is to poll the DOM with a timeout. Is there a better solution to be add with something like `executeScript`? You could run `requestAnimationFrame`, and then poll for an indicator that the click, etc. handler has indeed finis…

Ruby's Capybara encapsulates Selenium and waits until elements appear on the page (the default timeout is 2 seconds). So you can write simple sequential code like click_link('bar') expect(page).to have_content('baz') and it will work even if the baz element is injected into the page by an Ajax request to the server triggered by clicking on bar. I've been using it for many years but I didn't check how they implement i…

According to that documentation you linked, it just polls until `default_max_wait_time` (which defaults to 2 seconds).

Re: How to make Selenium tests reliable, scalable, and maintainable

#55
post #17

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

>tests that run for 40 minutes

This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.

Re: How to make Selenium tests reliable, scalable, and maintainable

#56

  > getWithRetry takes a function with a return value
  > 
  >   def numberOfChildren(implicit user: LucidUser): Int = {
  >    getWithRetry() {
  >      user.driver.getCssElement(visibleCss).children.size
  >    }
  >   }
  > 
  > predicateWithRetry takes function that returns a boolean and will retry on any false values
  > 
  >   def onPage(implicit user: LucidUser): Boolean = {
  >    predicateWithRetry() {
  >      user.driver.getCurrentUrl.contains(pageUrl)
  >    }
  >   }
At first I didn't get the difference between `getWithRetry` and `predicateWithRetry`, but then I noticed that the former throws an exception whereas the latter returns false. I infer that `getWithRetry` will handle exceptions thrown by the retried function.

In stb-tester[1] (a UI tool/framework targeted more at consumer electronics devices where the only access you have to the system-under-test is an HDMI output) after a few years we've settled on a `wait_until` function, which waits until the retried function returns a "truthy" value. `wait_until` returns whatever the retried function returns:

  def miniguide_is_up():
      return match("miniguide.png")

  press(Key.INFO)
  assert wait_until(miniguide_is_up)
  # or:
  if wait_until(miniguide_is_up): ...
(This is Python code.)

Since we use `assert` instead of throwing exceptions in our retried function, `wait_until` seems to fill both the roles of `getWithRetry` and `predicateWithRetry`. I suppose that you've chosen to go with 2 separate functions because so many of the APIs provided by Selenium throw exceptions instead of returning true/false.

  > doWithRetry takes a function with no return type
  >
  >   def clickFillColorWell(implicit user: LucidUser) {
  >    doWithRetry() {
  >      user.clickElementByCss("#fill-colorwell-color-well-wrapper")
  >    }
Unlike Selenium, when testing the UI of an external device we have no way of noticing whether an action failed, other than by checking the device's video output. For example we have `press` to send an infrared signal ("press a button on the remote control"), but that will never throw unless you've forgotten to plug in your infrared emitter. I haven't come up with a really natural way of specifying the retry of actions. We have `press_until_match`, but that's not very general. The best I have come up with is `do_until`, which takes two functions: The action to do, and the predicate to say whether the action succeeded.

  do_until(
      lambda: press(Key.INFO),
      miniguide_is_up)
It's not ideal, given the limitations around Python's lambdas (anonymous functions). Using Python's normal looping constructs is also not ideal:

  # Could get into an infinite loop if the system-under-test fails
  while not miniguide_is_up():
      press(Key.INFO)

  # This is very verbose, and it uses an obscure Python feature: `for...else`[2]
  for _ in range(10):
      press(Key.INFO)
      if miniguide_is_up():
          break
  else:
      assert False, "Miniguide didn't appear after pressing INFO 10 times"
Thanks for the article, I enjoyed it and it has reminded me to write up more of my experiences with UI testing. I take it that the article's sample code is Scala? I like its syntax for anonymous functions.

[1] http://stb-tester.com [2] https://docs.python.org/2/reference/compound_stmts.html#the-...

Re: How to make Selenium tests reliable, scalable, and maintainable

#57

> getWithRetry takes a function with a return value > > def numberOfChildren(implicit user: LucidUser): Int = { > getWithRetry() { > user.driver.getCssElement(visibleCss).children.size > } > } > > predicateWithRetry takes function that returns a boolean and will retry on any false values > > def onPage(implicit user: LucidUser): Boolean = { > predicateWithRetry() { > user.driver.getCurrentUrl.contains(pageUrl) > } >…

Thanks for the comment. We actually originally had a waitUntil function that was basically used for all three of the cases I mentioned above. In some sections of the code, it was just there to eat errors, other sections get some text, and yet others it was wrapped in an assert and needed to return a boolean. This led to chronic misuse around the code (I found 4-5 tests that simply forgot to wrap it in an assert effectively rendering the test completely worthless). The main benefit we got from splitting the methods out was making it clear to developers what it did. Catching all the exceptions thrown by Selenium instead of returning booleans was just an added benefit.

And you are correct, we are using Scala. There are some really cool things about the language, case classes, pattern matching, first order functions, and traits just to name a few.

Re: How to make Selenium tests reliable, scalable, and maintainable

#58

> getWithRetry takes a function with a return value > > def numberOfChildren(implicit user: LucidUser): Int = { > getWithRetry() { > user.driver.getCssElement(visibleCss).children.size > } > } > > predicateWithRetry takes function that returns a boolean and will retry on any false values > > def onPage(implicit user: LucidUser): Boolean = { > predicateWithRetry() { > user.driver.getCurrentUrl.contains(pageUrl) > } >…

Thanks for the comment. We actually originally had a waitUntil function that was basically used for all three of the cases I mentioned above. In some sections of the code, it was just there to eat errors, other sections get some text, and yet others it was wrapped in an assert and needed to return a boolean. This led to chronic misuse around the code (I found 4-5 tests that simply forgot to wrap it in an assert effectively rendering the test completely worthless). The main benefit we got from splitting the methods out was making it clear to developers what it did. Catching all the exceptions thrown by Selenium instead of returning booleans was just an added benefit.

And you are correct, we are using Scala. There are some really cool things about the language, case classes, pattern matching, first order functions, and traits just to name a few.

Re: How to make Selenium tests reliable, scalable, and maintainable

#59
post #27
post #12

Earlier quoted context omitted.

I had a Rails consultancy (Makandra) recently work on a JS-heavy application that I happen to own, and they got Selenium singing on it, which had been beyond my capabilities for years. One of their tricks, which you can inspect the implementation of in their (public) utilities library [+], is using basically a vendored Firefox per project and VNCing into that Firefox to drive things around. It is thus off-screen and…

I had been using Firefox driver in Xvfb but wasn't happy with the performance/stability. So I built a Selenium driver out of Java only (using JavaFX's embedded WebKit) and used a headless JRE windowing toolkit (Monocle). My project is still a pre-release but the headless capability, Java-only system requirement, and its ajax handling might make it useful to some people currently: https://github.com/MachinePublishers/…

Neat, but Affero public license? Ick.

Re: How to make Selenium tests reliable, scalable, and maintainable

#60
I currently manage a rather large test suite (around 700 different tests) using Selenium, which is all written in Ruby and Rspec (although I've also used Cucumber), and uses the gems Capybara (an abstraction layer for querying and manipulating the web browser via the Selenium driver) and SitePrism (for managing page objects and organizing re-usable sections).

The entire suite runs in around 10 minutes on CircleCI, using 8 parallel threads (each running an instance of the Firefox Selenium driver), and it is rock solid stable.

It took us a while to get to this point, though.

The hard part is handling timing due to Javascript race conditions on the front-end. I had to write my own helper methods like "wait_for_ajax" that I sprinkle in various page object methods to wait for any jQuery AJAX requests to complete. I also use a "wait_until_true" method that can evaluate a block of code over and over until a time limit has been reached before throwing an exception. Once you figure out ways to solve those types of issues, testing things with Selenium becomes a lot more stable and easy.

I have also used the exact same techniques (page objects, custom waiter methods for race conditions, etc) to test mobile apps on iOS and Android with Selenium.

It can be a challenge, but once you have a system down and you know what you are doing, it's not so bad.

Post reply on HN