Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make Selenium tests reliable, scalable, and maintainable

lucidchart.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: How to make Selenium tests reliable, scalable, and maintainable

#61

Post by EdwardDiego »

karlosmid wrote: post #26
EdwardDiego wrote: post #22

Earlier quoted context omitted.

Improve them how though? Speed? Reliability? If it's just a nicer API, that's all well and good, but until the key problems I face with Selenium are solved (slow and non-deterministic tests) then a nicer API to it is just rearranging deck-chairs on the Titanic.

It seems that you try to use selenium 2, or webdriver, in order to run your unit tests. Selenium is for browser test, and by its nature it can not run in milliseconds. Its execution time is in seconds. Even when use phantomjs webdriver. It is integration testing approach because it combines execution of several javascript modules. That run in real browser. Selenium has its purpose, but fast test execution is not one…

> It seems that you try to use selenium 2, or webdriver, in order to run your unit tests.

Nope. Integration tests. But integration tests that start a Firefox instance from scratch and have to be rerun multiple times to pass due to non-determinism are slow.

Re: How to make Selenium tests reliable, scalable, and maintainable

#63

Post by crdoconnor »

drothlis wrote: post #52
crdoconnor wrote: post #31

Earlier quoted context omitted.

>Is there a better solution Yes. And it's pretty simple: WebDriver driver = new FirefoxDriver(); driver.get("http://somedomain/url_that_delays_loading"); WebElement myDynamicElement = (new WebDriverWait(driver, 10)) .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement"))); From : http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp

I'm not sure this satisfies your parent poster's requirement of: "if it fails, you know about it pretty soon, without the need for long timeouts."

Well, you need to have a timeout.

You can make the timeout shorter when running the test on a dev environment, though, so you get quicker feedback about errors.

Re: How to make Selenium tests reliable, scalable, and maintainable

#64

Post by ericb »

ericb wrote: post #59
logn wrote: post #27

Earlier quoted context omitted.

I had been using Firefox driver in Xvfb but wasn't happy with the performance/stability. So I built a Selenium driver out of Java only (using JavaFX's embedded WebKit) and used a headless JRE windowing toolkit (Monocle). My project is still a pre-release but the headless capability, Java-only system requirement, and its ajax handling might make it useful to some people currently: https://github.com/MachinePublishers/…

Neat, but Affero public license? Ick.

To clarify--this was something I was interested in helping with, until I saw the license.

Re: How to make Selenium tests reliable, scalable, and maintainable

#65

Post by karlosmid »

EdwardDiego wrote: post #61
karlosmid wrote: post #26

Earlier quoted context omitted.

It seems that you try to use selenium 2, or webdriver, in order to run your unit tests. Selenium is for browser test, and by its nature it can not run in milliseconds. Its execution time is in seconds. Even when use phantomjs webdriver. It is integration testing approach because it combines execution of several javascript modules. That run in real browser. Selenium has its purpose, but fast test execution is not one…

> It seems that you try to use selenium 2, or webdriver, in order to run your unit tests. Nope. Integration tests. But integration tests that start a Firefox instance from scratch and have to be rerun multiple times to pass due to non-determinism are slow.

Could you please provide one example of non-determinism? I would like to understand what exactly do YOU mean by that term.

Re: How to make Selenium tests reliable, scalable, and maintainable

#66

Post by drothlis »

jarr416 wrote: post #57
drothlis wrote: post #56

> getWithRetry takes a function with a return value > > def numberOfChildren(implicit user: LucidUser): Int = { > getWithRetry() { > user.driver.getCssElement(visibleCss).children.size > } > } > > predicateWithRetry takes function that returns a boolean and will retry on any false values > > def onPage(implicit user: LucidUser): Boolean = { > predicateWithRetry() { > user.driver.getCurrentUrl.contains(pageUrl) > } >…

Thanks for the comment. We actually originally had a waitUntil function that was basically used for all three of the cases I mentioned above. In some sections of the code, it was just there to eat errors, other sections get some text, and yet others it was wrapped in an assert and needed to return a boolean. This led to chronic misuse around the code (I found 4-5 tests that simply forgot to wrap it in an assert effec…

> This led to chronic misuse around the code (I found 4-5 tests that simply forgot to wrap it in an assert effectively rendering the test completely worthless).

Yes, I've been bitten by that too -- it's too easy to forget the "assert". This morning it occurred to me that I could write a pylint (static analysis) checker to catch that, so I've done just that: https://github.com/stb-tester/stb-tester/commit/5e5bdbb

Re: How to make Selenium tests reliable, scalable, and maintainable

#67

Post by logn »

ericb wrote: post #64
ericb wrote: post #59

Earlier quoted context omitted.

Neat, but Affero public license? Ick.

To clarify--this was something I was interested in helping with, until I saw the license.

Is there any flavor of GPL license you would prefer more? I don't plan on BSD or Apache license in the foreseeable future.

Re: How to make Selenium tests reliable, scalable, and maintainable

#68

Post by logn »

boundlessdreamz wrote: post #39
logn wrote: post #27

Earlier quoted context omitted.

I had been using Firefox driver in Xvfb but wasn't happy with the performance/stability. So I built a Selenium driver out of Java only (using JavaFX's embedded WebKit) and used a headless JRE windowing toolkit (Monocle). My project is still a pre-release but the headless capability, Java-only system requirement, and its ajax handling might make it useful to some people currently: https://github.com/MachinePublishers/…

This looks very interesting! 1. How does it compare with phantomJS? 2. What's the current webkit version? 3. How often does the javaFX webkit update?

1. Not quite sure. I've only used PhantomJS via Selenium Ghost Driver. From that usage they're similar. The main difference is that my driver uses only Java so under the hood the JRE is launching WebKit through JNI and everything runs in the same JRE process.

2. Current WebKit version depends on the JRE used. Oracle Java 1.8.0_45 has WebKit version 537.44.

3. Java maintainers will update WebKit periodically, including within a major version. E.g., here they update WebKit for the 1.8.0_60 JRE: http://openjdk.java.net/jeps/239 ... Other than that I'm not sure.

Re: How to make Selenium tests reliable, scalable, and maintainable

#69

Post by stevebmark »

technion wrote: post #21
stevebmark wrote: post #11

Selenium tests are inherently slow, unreliable and flappy. They have been the bane of developers for every employer I've had. Do yourself a favor and write React and test your components without a browser driver in good ol' JS with the occasional JSDom shim. It removes almost the entire need for Selenium, which should be reserved for only the faintest of smoke tests. And please, if you have to use Selenium, use headl…

Can you expand on this? The extent of the testing I'm currently interested in is "load a page, does the JS on that page run without error"? It won't execute from a CLI and everyone I talked to pointed me at Selenium.

Testing for a page load without JS error is a fine use case, and is an example of what I meant by the "faintest of smoke tests." It's a test that has very little chance to flap, fail, or force you to write hacky commands around Selenium's unreliable API.

Re: How to make Selenium tests reliable, scalable, and maintainable

#70

Post by stevebmark »

omouse wrote: post #55
jmickey wrote: post #17

Earlier quoted context omitted.

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable. We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

> tests that run for 40 minutes This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.

40 minutes from clicking a button to deploy is actually abysmal, especially when you need to worry about things like rollbacks, or deploying at the end of the day, or releasing quick hotfixes to users. In modern build processes, even 10 minutes seems too long.
Post reply on HN