Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our team's troubles writing automated UI tests

medium.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Our team's troubles writing automated UI tests

#11

Post by lostintangent »

benologist wrote: post #8

I solved my UI tests by scripting a ton of screenshots for my documentation using Puppeteer, going through each thing the user can do and proving they can do it at multiple device resolutions. It was a lot of work but now it is quite simple - await owner.click('Create plan') await owner.screenshot('Fill out form') One thing I did wrong that I have to revisit is I don't confirm the screenshots hold any data I'm expect…

I’d love to hear more about this! Do you have automation that runs as part of your CI builds, and dynamically captures the screenshots for your documentation using Puppeteer? If so, that is so cool :)

Have you had much trouble with the automation scripts breaking as a result of app churn?

Re: Our team's troubles writing automated UI tests

#12

Post by arjun27 »

legulere wrote: post #7

At the project I’m currently working on we evaluated WinAppDriver (for WPF) and found it to be pretty buggy, slow and low featured. In the end we decided for a commercial product on around the same level as it, but if you’re on windows you can always use the ui automation Framework.

Any recommendations of commercial products that worked for you?

Re: Our team's troubles writing automated UI tests

#14

Post by benologist »

lostintangent wrote: post #11
benologist wrote: post #8

I solved my UI tests by scripting a ton of screenshots for my documentation using Puppeteer, going through each thing the user can do and proving they can do it at multiple device resolutions. It was a lot of work but now it is quite simple - await owner.click('Create plan') await owner.screenshot('Fill out form') One thing I did wrong that I have to revisit is I don't confirm the screenshots hold any data I'm expect…

I’d love to hear more about this! Do you have automation that runs as part of your CI builds, and dynamically captures the screenshots for your documentation using Puppeteer? If so, that is so cool :) Have you had much trouble with the automation scripts breaking as a result of app churn?

So far I still manually trigger a regeneration but it's only a matter of time till it just runs automatically on any update.

The main problem has been with Puppeteer, there is a chance it will just randomly crash at any point and it feels like there are a million little race conditions where you need to wait or keep retrying to access an element after it says a page has loaded so I have a lot of ugly code in my puppeteer-helper like:

    while (true) {
        try { 
            element = await active.$('#element')
        } catch (error) {
        }
        if (!element) {
           sleep(100)
           continue
        }
        break
    }
Another issue only pertained to getting the simulated device screenshots, switching the viewport configuration reloads the page which for me makes any state like "yes, you just deleted x" be an "x does not exist" error so at the moment I get around that by doing each device independently which takes a fair bit longer where slow APIs are involved.

This is one of the 'recipes' -

https://github.com/userdashboard/userdashboard.github.io/blo... with some glue for Puppeteer v1 so a bit WIP still - https://github.com/userdashboard/userdashboard.github.io/blo...

The end result I think is really cool:

https://userdashboard.github.io/administrators/stripe-subscr...

In the final incarnation people using my software will be able to point the screenshot generator at their own website and regenerate all the screenshots so they can reuse the documentation.

Re: Our team's troubles writing automated UI tests

#15

Post by karldcampbell »

Macha wrote: post #9
mavelikara wrote: post #4

Earlier quoted context omitted.

After reading through the website and watching their video, I still don't understand how it works out what it's value proposition is. Is there some demo where I can see and understand how Cypress provides value over the state of the art?

It solves the same problem as Selenium without actually being Selenium. The one time Selenium was reliable in my company was when our team had a dedicated, massively overprovisioned grid. The biggest pro is it just has less random "Oh, guess that timed out?" issues than when we used Selenium. The biggest negative is it's chrome only.

Have you looked into TestCafe[1]? I've only used it for a PoC, but it has better browser support than Cypress.

[1]https://devexpress.github.io/testcafe/

Re: Our team's troubles writing automated UI tests

#16

Post by zmmmmm »

I think this is one of the "dirty secrets" of current software development ... one of those things that pretty much nobody does very well (at least, anywhere I have seen).

My view is that we won't make real progress on proper UI testing until there is a paradigm shift that moves responsibility for creating applications that are fundamentally testable to developers. This is not dissimilar to how the idea of "devops" finally recognised that managing deployments and infrastructure is actually first class development activity.

The problem is that testing a system that has not been designed for it is almost fundamentally intractable. I have seen toolkits that auto-generate random ids for every element on the page so that it is almost impossible to hook onto anything within the page as a reliable anchor to identify testable points within the page. Even when those things do exist it is usually only incidental and not an agreed contract b/w developer and test automation engineer. So they will always break unexpectedly etc etc.

Re: Our team's troubles writing automated UI tests

#17

Post by jchb »

Article should really be called “our teams troubles writing automated UI tests for native windows apps”.

The article mentions “If you follow the Page Object Model pattern, then for each page and control in your application, you create models so your tests can find and interact with the elements on that page or control.“.

Not being a windows programmer I am not familiar with this model in detail, but a major issue seems to be that the UI toolkit doesn’t have any built in support for UI automation.

For example on iOS and macOS, the accessibility system (which is what e.g the screen reader for blind users use to drive the application) is repurposed for UI tests. This means that if your app is accesible, which it pretty much is by default where you use standard UI controls, with minor code changes such as assigning unique identifiers to UI elements where needed, it is also UI-testable.

Re: Our team's troubles writing automated UI tests

#18

Post by _____s »

jchb wrote: post #17

Article should really be called “our teams troubles writing automated UI tests for native windows apps ”. The article mentions “If you follow the Page Object Model pattern, then for each page and control in your application, you create models so your tests can find and interact with the elements on that page or control.“. Not being a windows programmer I am not familiar with this model in detail, but a major issue se…

The Page Object pattern is common in Ruby (and some JS frameworks, like ember.js) too. I agree that the issues mentioned in this article seem mostly windows app centric. For JS apps running in the browser, the testing story is pretty good these days.

Re: Our team's troubles writing automated UI tests

#19

Post by pjmlp »

jchb wrote: post #17

Article should really be called “our teams troubles writing automated UI tests for native windows apps ”. The article mentions “If you follow the Page Object Model pattern, then for each page and control in your application, you create models so your tests can find and interact with the elements on that page or control.“. Not being a windows programmer I am not familiar with this model in detail, but a major issue se…

WPF and UWP support UI Automation, there are even VS project templates for. Forms does not, beyond the typical Win32 SendMessage().

However there are good UI tooling that does a good job in spite of it, assuming that one is willing to pay for them.

https://www.ibm.com/us-en/marketplace/rational-functional-te...

https://www.ranorex.com/

https://eggplant.io/

https://www.tricentis.com/

https://www.microfocus.com/en-us/products/unified-functional...

Re: Our team's troubles writing automated UI tests

#20

Post by mping »

mavelikara wrote: post #4
GvS wrote: post #2

I used to have troubles with UI tests too, but then I discovered Cypress ( https://www.cypress.io/ ). You can still run into an issue once in a while, but it's way better than anything else and it's getting even better with each release.

After reading through the website and watching their video, I still don't understand how it works out what it's value proposition is. Is there some demo where I can see and understand how Cypress provides value over the state of the art?

Cypress aims to be as deterministic as possible. You can write tests that wait for a specific xhr request or even mock them. The combination of that and the promise based API makes it much more reliable than selenium.

In practice, I have had much less flakiness in my Cypress tests than with selenium or other webdriver API.

Post reply on HN