Show HN: Headless Chrome Crawler

#1

Post by yujiosaka » Thu, Feb 22, 2018, 12:31 PM UTC

Show HN: Headless Chrome Crawler
github.com

Re: Show HN: Headless Chrome Crawler

#2

Post by bryanrasmussen » Thu, Feb 22, 2018, 2:08 PM UTC

Can't see from examples, how do I get back individual elements from the body?

Re: Show HN: Headless Chrome Crawler

#3

Post by bryanrasmussen » Thu, Feb 22, 2018, 2:26 PM UTC

also how does this handle pages that load with a small number of links and then uses JS to write in a bunch of DOM nodes and links?

Re: Show HN: Headless Chrome Crawler

#4

Post by diggan » Thu, Feb 22, 2018, 2:27 PM UTC

bryanrasmussen wrote: post #2

Can't see from examples, how do I get back individual elements from the body?

Not a user of this tool, but https://github.com/yujiosaka/headless-chrome-crawler#event-n... points to https://github.com/GoogleChrome/puppeteer/blob/master/docs/a... where you can grab elements.

Basically, when new page event happens, you get the `page` object where you have access to it and can do queries.

Re: Show HN: Headless Chrome Crawler

#5

Post by ptasker » Thu, Feb 22, 2018, 2:33 PM UTC

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily.

https://github.com/GoogleChrome/puppeteer

Re: Show HN: Headless Chrome Crawler

#6

Post by trevyn » Thu, Feb 22, 2018, 2:45 PM UTC

bryanrasmussen wrote: post #3

also how does this handle pages that load with a small number of links and then uses JS to write in a bunch of DOM nodes and links?

I don't know about this project specifically, but typically with headless Chrome, you let it run the JS and then read the DOM.

Re: Show HN: Headless Chrome Crawler

#7

Post by isuckatcoding » Thu, Feb 22, 2018, 2:51 PM UTC

ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Yeah I’d really rather that people made extensions to Pupeteer rather than a whole new library.

Re: Show HN: Headless Chrome Crawler

#8

Post by robk » Thu, Feb 22, 2018, 2:53 PM UTC

ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Puppeteer seems needlessly difficult to use on a VPS. I'd prefer an easily dockerized version but there seems to be nothing robust and they make it VERY hard to connect to a docker instance just running Chrome for the websocket/9222 interface sadly.

Re: Show HN: Headless Chrome Crawler

#9

Post by ej12n » Thu, Feb 22, 2018, 3:02 PM UTC

diggan wrote: post #4

bryanrasmussen wrote: post #2

Can't see from examples, how do I get back individual elements from the body?

Not a user of this tool, but https://github.com/yujiosaka/headless-chrome-crawler#event-n... points to https://github.com/GoogleChrome/puppeteer/blob/master/docs/a... where you can grab elements. Basically, when new page event happens, you get the `page` object where you have access to it and can do queries.

`page.$(query)` or `page.$$(queryAll)` to be more specific

https://github.com/GoogleChrome/puppeteer/blob/v1.1.0/docs/a...

Re: Show HN: Headless Chrome Crawler

#10

Post by hartator » Thu, Feb 22, 2018, 3:18 PM UTC

I was stuck last time I was using headless chrome when I needed to use a proxy with an username and a paasword. Headless chrome just doesn't support it. Any changes on that?