Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Headless Chrome Crawler

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Show HN: Headless Chrome Crawler

#31

Post by RossM »

robk wrote: post #8
ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Puppeteer seems needlessly difficult to use on a VPS. I'd prefer an easily dockerized version but there seems to be nothing robust and they make it VERY hard to connect to a docker instance just running Chrome for the websocket/9222 interface sadly.

I've done this recently actually. Take a look at the yukinying/chrome-headless-browser[0] image. You'll need to run with the SYS_ADMIN capability and up the shm_size to 1024M (you can workaround the SYS_ADMIN cap with a seccomp file but I didn't have much luck with that). Other than that oddness it works pretty well (and with Puppeteer 1.0, with far fewer crashes).

[0]: https://github.com/yukinying/chrome-headless-browser-docker

Re: Show HN: Headless Chrome Crawler

#32

Post by bryanrasmussen »

diggan wrote: post #4
bryanrasmussen wrote: post #2

Can't see from examples, how do I get back individual elements from the body?

Not a user of this tool, but https://github.com/yujiosaka/headless-chrome-crawler#event-n... points to https://github.com/GoogleChrome/puppeteer/blob/master/docs/a... where you can grab elements. Basically, when new page event happens, you get the `page` object where you have access to it and can do queries.

yes I've used puppeteer, but I couldn't see where it exposed the page object, but looking closer I saw

HCCrawler.launch({ // Function to be evaluated in browsers evaluatePage: (() => ({ title: $('title').text(), })), // Function to be called with evaluated results from browsers onSuccess: (result => { console.log(result); }), })

which doesn't look that good for my other worry, that you come to a dynamic JS built up page, load the DOM and evaluate in some milliseconds and then that DOM is changed after you've done your incorrect evaluation.

Re: Show HN: Headless Chrome Crawler

#33

Post by asadlionpk »

robk wrote: post #8
ptasker wrote: post #5

Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily. https://github.com/GoogleChrome/puppeteer

Puppeteer seems needlessly difficult to use on a VPS. I'd prefer an easily dockerized version but there seems to be nothing robust and they make it VERY hard to connect to a docker instance just running Chrome for the websocket/9222 interface sadly.

I recently did this in a Docker.

Let me quickly add instructions here, first you need to install some dependancies, add the following to dockerfile:

  RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
Secondly, launch puppeteer with --no-sandbox option:

  browser = await puppeteer.launch({
      args: ['--no-sandbox'] /*, headless: false*/
    })
That should do it.

Re: Show HN: Headless Chrome Crawler

#34

Post by bryanrasmussen »

trevyn wrote: post #6
bryanrasmussen wrote: post #3

also how does this handle pages that load with a small number of links and then uses JS to write in a bunch of DOM nodes and links?

I don't know about this project specifically, but typically with headless Chrome, you let it run the JS and then read the DOM.

most naive code I see is like the following

    const page = await crawler.browser.newPage();
    await page.goto(url);
    await page.waitForSelector("a[href]");
    const hrefs = await page.evaluate(
    () => Array.from(document.body.querySelectorAll('a[href]'), ({ href }) => href)
    );
and then you do something with hrefs.

However if you have a page that loads with 4 links defined, does its script and then ends up 100+ links you miss the 100+ links. I notice people often failing to account for this in their crawlers, so I wondered if this one was.

Re: Show HN: Headless Chrome Crawler

#35

Post by princehonest »

I've been considering writing my own puppeteer docker image such that one could freeze the image at crawl time after a page has loaded. This would allow me to re-write the page-parsing logic after the page layout changes. Has anyone done this already or know of any other efforts to serialize the puppeteer page object to handle parsing bugs?

Re: Show HN: Headless Chrome Crawler

#36

Post by nikisweeting »

I'm thinking about adding a crawler to Bookmark Archiver, to augment the headless chrome screenshotting and PDFing that it already does.

Wget is also a pretty robust crawler, but people have requested a proxy that archives every site they visit in real-time more than a crawler.

Re: Show HN: Headless Chrome Crawler

#37

Post by transreal »

hartator wrote: post #10

I was stuck last time I was using headless chrome when I needed to use a proxy with an username and a paasword. Headless chrome just doesn't support it. Any changes on that?

Actually, we just figured out how to do this. Details here: https://bugs.chromium.org/p/chromium/issues/detail?id=741872...

Re: Show HN: Headless Chrome Crawler

#38

Post by hartator »

transreal wrote: post #37
hartator wrote: post #10

I was stuck last time I was using headless chrome when I needed to use a proxy with an username and a paasword. Headless chrome just doesn't support it. Any changes on that?

Actually, we just figured out how to do this. Details here: https://bugs.chromium.org/p/chromium/issues/detail?id=741872...

Awesome. I need to figure out a way to make it work with our Ruby code, but it shouldn't be that hard. Thanks.

Re: Show HN: Headless Chrome Crawler

#39

Post by radioo75555 »

princehonest wrote: post #35

I've been considering writing my own puppeteer docker image such that one could freeze the image at crawl time after a page has loaded. This would allow me to re-write the page-parsing logic after the page layout changes. Has anyone done this already or know of any other efforts to serialize the puppeteer page object to handle parsing bugs?

In large scale scraping they always separate loading pages from processing the data. Easiest thing would be to wait for the page to load and then save the html
Post reply on HN