Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Headless Chrome Crawler

github.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Show HN: Headless Chrome Crawler

#13

Post by hartator »

jancurn wrote: post #11
hartator wrote: post #10

I was stuck last time I was using headless chrome when I needed to use a proxy with an username and a paasword. Headless chrome just doesn't support it. Any changes on that?

There's a workaround - https://blog.apify.com/how-to-make-headless-chrome-and-puppe...

Thanks. Wish it was simpler. It seems overkill to have to an extra proxy in the middle with no auth to authenticate with the one with auth, just to make headless chrome working.

Re: Show HN: Headless Chrome Crawler

#14

Post by timstapl »

hartator wrote: post #13
jancurn wrote: post #11

Earlier quoted context omitted.

There's a workaround - https://blog.apify.com/how-to-make-headless-chrome-and-puppe...

Thanks. Wish it was simpler. It seems overkill to have to an extra proxy in the middle with no auth to authenticate with the one with auth, just to make headless chrome working.

There's also page.authenticate, which has worked well for me. https://github.com/GoogleChrome/puppeteer/blob/master/docs/a...

Re: Show HN: Headless Chrome Crawler

#15

Post by jancurn »

hartator wrote: post #13
jancurn wrote: post #11

Earlier quoted context omitted.

There's a workaround - https://blog.apify.com/how-to-make-headless-chrome-and-puppe...

Thanks. Wish it was simpler. It seems overkill to have to an extra proxy in the middle with no auth to authenticate with the one with auth, just to make headless chrome working.

You can also use page.authenticate() for that - see a note at the bottom of the article. Also see https://github.com/GoogleChrome/puppeteer/pull/1732

Re: Show HN: Headless Chrome Crawler

#16

Post by itsjustme2 »

ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Looks like this is actually built on top of puppeteer. See the "Note" under "Installation": https://github.com/yujiosaka/headless-chrome-crawler/blob/ma...

Re: Show HN: Headless Chrome Crawler

#17

Post by tesin »

While as a developer I find this super interesting, as a system administrator this makes me cringe. We don't have a lot of resources for servers, and I end up spending a disproportionate amount of time banning IPs from bots running poorly configured tools like this, which aren't rate limited and crush websites.

I'm grateful that "Obey robots.txt" is listed as part of it's standard behavior. If only scrapers cared enough to use it as well.

Re: Show HN: Headless Chrome Crawler

#18

Post by walterstucco »

hartator wrote: post #10

I was stuck last time I was using headless chrome when I needed to use a proxy with an username and a paasword. Headless chrome just doesn't support it. Any changes on that?

I created a tiny docker image to solve this issue

It's around 200k and very easy to configure

https://hub.docker.com/r/massimo/cntlm/

Re: Show HN: Headless Chrome Crawler

#20

Post by codedokode »

This has been possible for a long time with any browser using Selenium for example. It has APIs and client libraries for many languages.

Also using a real browser brings a lot of problems: high resource consumption, hangs, it is unclear when the page has finished loading etc. You have to supervise all browser processes. And if you use promises, there is high chance that you will miss error messages because promises hide them by default.

Post reply on HN