Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Headless Chrome Crawler

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Show HN: Headless Chrome Crawler

#21

Post by codedokode »

tesin wrote: post #17

While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up spending a disproportionate amount of time banning IPs from bots running poorly configured tools like this, which aren't rate limited and crush websites. I'm grateful that "Obey robots.txt" is listed as part of it's standard behavior. If only scrapers cared eno…

Such crawler should not be difficult to ban by looking at stats - if there are many requests per IP per unit of time, or many requests from data center IPs, or many requests from Linux browsers, it is likely bots and you can ban them (you can ban whole data center to be sure).

Re: Show HN: Headless Chrome Crawler

#23

Post by superasn »

tesin wrote: post #17

While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up spending a disproportionate amount of time banning IPs from bots running poorly configured tools like this, which aren't rate limited and crush websites. I'm grateful that "Obey robots.txt" is listed as part of it's standard behavior. If only scrapers cared eno…

I've found that mod_evasive[1] works particularly well in these situation and helped us a lot dealing with it (though I'm not a sysadmin and I'm sure there are better tools to deal with it). But for someone who is just a webmaster, I'd recommend using it for a quick dirty fix for such hassles.

[1] https://www.digitalocean.com/community/tutorials/how-to-prot...

Re: Show HN: Headless Chrome Crawler

#24

Post by chatmasta »

ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Puppeteer has some limitations. You can’t install extensions, for example.

I haven’t looked into it, but I imagine it has a pretty clear fingerprint as well. So it would be easier to block than stock chrome in headless mode.

Re: Show HN: Headless Chrome Crawler

#25

Post by tegansnyder »

There are a lot of folks reevaluating their crawling engines lately now that Chrome headless is maturing. To me there are some important considerations in terms of CPU/memory footprint that go into distributing a large headless crawling architecture.

The stuff we are not seeing open-sourced is the solutions companies are building around trimmed down specialized versions of the headless browsers like Chrome headless, Servo, Webkit. People are running distributed versions of these headless browsers using Apache Mesos, Kubernetes, and Kafka queues.

Re: Show HN: Headless Chrome Crawler

#28

Post by simooooo »

chatmasta wrote: post #24
ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Puppeteer has some limitations. You can’t install extensions, for example. I haven’t looked into it, but I imagine it has a pretty clear fingerprint as well. So it would be easier to block than stock chrome in headless mode.

I can't puppeteer to screen capture websites very well, crashes entirely after a few

Re: Show HN: Headless Chrome Crawler

#29

Post by webfolder »

codedokode wrote: post #20

This has been possible for a long time with any browser using Selenium for example. It has APIs and client libraries for many languages. Also using a real browser brings a lot of problems: high resource consumption, hangs, it is unclear when the page has finished loading etc. You have to supervise all browser processes. And if you use promises, there is high chance that you will miss error messages because promises h…

Selenium is not suitable for website crawling. cdp4j is more suitable for this kind of works.

https://github.com/webfolderio/cdp4j

Re: Show HN: Headless Chrome Crawler

#30

Post by kodablah »

chatmasta wrote: post #24
ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Puppeteer has some limitations. You can’t install extensions, for example. I haven’t looked into it, but I imagine it has a pretty clear fingerprint as well. So it would be easier to block than stock chrome in headless mode.

Unless something has changed that I missed, you can install extensions (I complained when the default args messed this up [0]). For example, I built something that uses puppeteer and an extension to capture audio and video of a tab [1]. It's just headless mode that doesn't allow extensions [2] (which I now realize is probably what you meant).

0 - https://github.com/GoogleChrome/puppeteer/issues/850 1 - https://github.com/cretz/chrome-screen-rec-poc/tree/master/a... 2 - https://bugs.chromium.org/p/chromium/issues/detail?id=706008

Post reply on HN