Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

github.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#71

Post by schickling »

inertial wrote: post #70

I'll just add some related projects I've used / tried in the past. The promise of fast execution time in parallel is tempting with Chromeless. Thanks for sharing. - https://github.com/webdriverio/webdriverio - https://github.com/nightwatchjs/nightwatch - https://github.com/assaf/zombie - https://github.com/dhamaniasad/HeadlessBrowsers

Thanks a lot for bringing this up. We've tried all of the projects listed above before we began to implement Chromeless.

Ultimately it was the combination of using headless Chrome and the ability to execute code in parallel on Lambda, which made us invest in Chromeless.

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#72

Post by pravda »

hackits wrote: post #65
pravda wrote: post #55

Earlier quoted context omitted.

>What are sites' motivations for blocking AWS IPs? I block AWS. So many crawlers up to so much nonsense! I don't block by IP, but by hostname. $block='.amazonaws.com'; $ua = @$_SERVER['HTTP_USER_AGENT']; if (stripos($rh,$block)!==false && stripos($ua,'Silk')===false && stripos($ua,'Safari')===false){ $block_visitor=true; $message="Blocked Host:Amazon Web Services"; }

Just curious what have you seen crawlers do to make you conclude they're up to nonsense?

Well, from amazonaws.com, there are so many requests for wp-login.php!

And then all the off-brand scraping companies use amazonaws.com.

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#73

Post by schickling »

ivansavz wrote: post #67
schickling wrote: post #2

I'm really excited to finally open-source Chromeless. We've used NightmareJS and similar tools before to run integration tests but these basically added ~20min to each build. With Chromeless we were able to reduce this time to under a minute! Here is btw a demo playground to try it out: https://chromeless.netlify.com/ Let me know if you have any questions :)

How difficult would it be to also support google cloud functions and the azure offerings? This seems like a really useful standard tool that lots of people might want to use. CI jobs on pull requests that take seconds instead of minutes = big win!

This should actually be pretty easy. There is no reason that this is bound to AWS Lambda. We're more than open to accept PRs that enable multi-cloud support.

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#74

Post by schickling »

titel wrote: post #39

@schickling - When will the PDF support arrive? https://github.com/graphcool/chromeless/blob/master/docs/api...

As I can't give you an exact estimate when it will arrive, seems like a lot of people are already asking for it here: https://github.com/graphcool/chromeless/issues/5#issuecommen...

Please feel more than welcome to take a stab at it yourself and create a PR for this feature! :)

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#75

Post by mrmondo »

afandian wrote: post #41

I've got the impression that lots of sites block AWS IP addresses. I wonder if this would hamper the practical use of this on Lambda. I'm doing something similar, and this concern was one motivation for running in our datacentre vs EC2. Does anyone have concrete info on rates of bots blocked from AWS IPs?

Indeed, so many of AWS’s IP ranges are used for DDoS and malicious behaviour they end up getting blacklisted due to their poor reputation. It’s a bit like using one third party resellers shared IP ranges in your mail relay - you’re asking to end up on reputation based lists. There’s a lot to be said for your IP reputation on the internet and when you outsource that - you outsource your freedom to maintain your reputation risk.

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#76

Post by kensoh »

This is a really cool project, but looking closer at the API and issues raised it seems that the features are being over-promised.

- "Do pretty much everything you've used PhantomJS, NightmareJS or Selenium for before".

The main features of those tools plus their ability to handle a large range of edge cases are built up over the years in production use and do not seem to be already in Chromeless. Also, Lambda costs can be a significant point of consideration for professional test automation with large volume.

Nevertheless, there's no turning back as flood gates have been opened and many developers are noticing Chromeless. I believe, with enough dedication from Chromeless maintainers, they may be able to channel the attention and contributions to shape Chromeless to be the main challenger to existing test automation approaches. That will really be a blessing to the open-source community!

The only catch I believe, is it may be easier for those existing tools to be made working in Lambda or implement a similar form of parallelism while still having their mature API, than for Chromeless to catch up to the state of maturity of those tools. But as they say, growth solves almost every problem, so issues like these may be ironed out through collaborative efforts from contributors/maintainers.

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#77

Post by afandian »

pravda wrote: post #72
hackits wrote: post #65

Earlier quoted context omitted.

Just curious what have you seen crawlers do to make you conclude they're up to nonsense?

Well, from amazonaws.com, there are so many requests for wp-login.php! And then all the off-brand scraping companies use amazonaws.com.

What do you mean by off-brand scraping? You mean search engines that you haven't heard of, or copyright violating orgs?

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#78

Post by afandian »

ingenuous2 wrote: post #60
afandian wrote: post #41

I've got the impression that lots of sites block AWS IP addresses. I wonder if this would hamper the practical use of this on Lambda. I'm doing something similar, and this concern was one motivation for running in our datacentre vs EC2. Does anyone have concrete info on rates of bots blocked from AWS IPs?

Fairly simple (if this is like the other programmatic headless browsers) to make a request through a proxy.

And where do you run the proxy?

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#79

Post by sorenbs »

pkilgore wrote: post #43
titel wrote: post #39

@schickling - When will the PDF support arrive? https://github.com/graphcool/chromeless/blob/master/docs/api...

Also interested, this would be excellent fit for a use case I have archiving certain important government websites. Btw, does the .viewport() option not work in the demo? I'm seeing a `TypeError: Failed to fetch` when I set one.

You need to do it like this:

https://chromeless.netlify.com/#src=const%20chromeless%20=%2...

Re: Show HN: Chromeless – Headless Chrome Automation on AWS Lambda

#80

Post by schickling »

kensoh wrote: post #76

This is a really cool project, but looking closer at the API and issues raised it seems that the features are being over-promised. - "Do pretty much everything you've used PhantomJS, NightmareJS or Selenium for before". The main features of those tools plus their ability to handle a large range of edge cases are built up over the years in production use and do not seem to be already in Chromeless. Also, Lambda costs…

Hi kensoh, thanks a lot for your great comment.

I totally agree with you! It took years for these tools to mature and so it will be the case for Chromeless. There are probably a range of edge-cases that yet have to be solved but like you said, I'm very optimistic that together with our great community, we'll be able to handle all of these cases.

The big incentive for us to create Chromeless instead of using Nightmare or similar (which I've done for years) was the fact that you can now use headless Chrome (which provides a way more stable foundation) + the ability to execute the code on AWS Lambda which solves the parallelisation question. I hope this makes sense to you :)

Post reply on HN