Live data from Hacker News

Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

github.com

31–39 of 39 posts

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#31
post #25
post #14

Two things: 1. Please do not test a web app with Chrome only, we don't want to go back to a world with a single browser 2. > So, until puppeteer supports this, we will rely on jsdom, a package available via npm JSDOM is not just a package on npm, it's an engineering piece of art

I have given up on Firefox. Every bug I submit takes it own sweet time to get fixed.

How about the bufixes you submit, are they faster?

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#33
I am writing almost the same thing but for PDF [1]. But I am having trouble with scaling.

I got able to make it run inside a docker.

In this exact moment the example at the repo is just returning a blank PDF but the problem is at the API Gateway.

[1] https://github.com/tecnospeed/pastor

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#34
post #30

Where I work we prefer jQuery to the native DOM API for scraping. It really speeds up the process of extracting data. For example with Puppeteer you can do page.injectFile("jquery-3.2.1.min.js"). I think that would simplify your evaluate() calls. It would also be easy to speed up the whole process by doing a single evaluate() call per page with all your scraping code in it. BTW we just released an article with tips &…

Good suggestion. I would update soon. Thank you

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#35
post #14

Two things: 1. Please do not test a web app with Chrome only, we don't want to go back to a world with a single browser 2. > So, until puppeteer supports this, we will rely on jsdom, a package available via npm JSDOM is not just a package on npm, it's an engineering piece of art

Just to note, there are other uses in browser automation beyond testing (this article is about webscraping). Selenium webdriver have its own limitations, and they aren't willing to add features to cover other use cases.

I hope Puppeteer become a standard.

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#36
post #2

> Since the official announcement of Chrome Headless, many of the industry standard libraries for automated testing have been discontinued by their maintainers. The prominent of these are PhantomJS and Selenium IDE for Firefox. Correct me if I'm wrong, but if I'm notm mistaken Selenium IDE has been discontinued due to lack of mantainers, and that has little if any relation to Chrome Headless. The IDE is just a more e…

Which is kind of sad. No Chrome Headless library supports dowloading of files.

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#37
post #22

Earlier quoted context omitted.

You should be able to use a data URI containing the HTML string

It's not allowed anymore: https://groups.google.com/a/chromium.org/forum/m/#!topic/bli...

That's not correct.

Initial assumption when reading the thread was that navigating to a data URI would be handled like entry of a data URI into the omnibox and still be allowed.

A small test case confirms that assumption - it works.

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#38
post #14

Two things: 1. Please do not test a web app with Chrome only, we don't want to go back to a world with a single browser 2. > So, until puppeteer supports this, we will rely on jsdom, a package available via npm JSDOM is not just a package on npm, it's an engineering piece of art

Headless coming to Firefox. I suspect Puppeteer would support it too very soon.
Post reply on HN