Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Automatically create QA environments for your GitHub pull requests

prune.io

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Show HN: Automatically create QA environments for your GitHub pull requests

#11

Post by 3am »

mpotter wrote: post #6
jawns wrote: post #5

Earlier quoted context omitted.

Can you give more details about Visual Regression Testing? If you're comparing screenshots, how do you account for dynamic content?

Sure! We use a web driver to simulate browsing in an actual client, so Javascript, etc. is executed as it normally would be.

WebDriver/Selenium? That's not really visual, like some kind of CV/OCR tool like Sikuli would be. It's a nice tool, I would be more inclined to use it if it supported more robust test environment provisioning (eg, if the pull request is a schema migration) and it gave more control over how to define my own tests, like Sauce Labs.

edit: never mind, I realized you're probably using the built in screenshot features with webdriver and just diffing the images.

Re: Show HN: Automatically create QA environments for your GitHub pull requests

#15

Post by mpotter »

jawns wrote: post #7
mpotter wrote: post #6

Earlier quoted context omitted.

Sure! We use a web driver to simulate browsing in an actual client, so Javascript, etc. is executed as it normally would be.

But what about content that changes on each page load, like a timestamp or a rotating banner ad? Would that be flagged as a regression?

Yeah, it's not perfect but right now we're solving this with tolerance thresholds on the image diffs.

Re: Show HN: Automatically create QA environments for your GitHub pull requests

#17

Post by maerek »

mpotter wrote: post #15
jawns wrote: post #7

Earlier quoted context omitted.

But what about content that changes on each page load, like a timestamp or a rotating banner ad? Would that be flagged as a regression?

Yeah, it's not perfect but right now we're solving this with tolerance thresholds on the image diffs.

Can you share any more details on how the image diffing is done? BBC has a github repo called Wraith that uses ImageMagick and PhantomJS to accomplish a similar task (without the awesomeness of on-demand testing environments). Always curious to learn more about how people are solving the GUI testing problem on the web.

Re: Show HN: Automatically create QA environments for your GitHub pull requests

#18

Post by mpotter »

michaelmior wrote: post #13

How do you manage to support whatever crazy environments a dev may cook up and all the services needed to get a working QA environment set up?

We'll announce our supported environment list shortly but the preview answer is: Docker & Heroku Buildpacks :)

Re: Show HN: Automatically create QA environments for your GitHub pull requests

#20

Post by dbarlett »

maerek wrote: post #17
mpotter wrote: post #15

Earlier quoted context omitted.

Yeah, it's not perfect but right now we're solving this with tolerance thresholds on the image diffs.

Can you share any more details on how the image diffing is done? BBC has a github repo called Wraith that uses ImageMagick and PhantomJS to accomplish a similar task (without the awesomeness of on-demand testing environments). Always curious to learn more about how people are solving the GUI testing problem on the web.

There's also Depicted (dpxdt)[1] - more workflow, Python instead of Ruby.

[1] https://github.com/bslatkin/dpxdt

Post reply on HN