Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#21

Post by testcross »

Do you know if it is possible to render a page without serving it from a web server? For example, I have the html of one page of my domain generated by a test. I would like to use puppeteer to render it. But I don't want to setup a http server for this. I would like to give a string with the html + a url to page.goto and let it render the page like it comes from the real server.

I guess I can cheat by intercepting the request and respond with the html I already have. But I wonder if there is already something existing.

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#22

Post by houli »

testcross wrote: post #21

Do you know if it is possible to render a page without serving it from a web server? For example, I have the html of one page of my domain generated by a test. I would like to use puppeteer to render it. But I don't want to setup a http server for this. I would like to give a string with the html + a url to page.goto and let it render the page like it comes from the real server. I guess I can cheat by intercepting th…

You should be able to use a data URI containing the HTML string

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#23

Post by tw21 »

escap wrote: post #7
jaxn wrote: post #4

Earlier quoted context omitted.

Related or not, it seems like a valid point. We switched to chrome headless after a post from thoughtbot made me question Capybara-WebKit's future.

Selenium IDE was discontinued due to the change of extension (from XPI to WebExtension) in Firefox. Nothing to do with Chrome Headless. see https://seleniumhq.wordpress.com/2017/08/09/firefox-55-and-s... and associated HN discussion https://news.ycombinator.com/item?id=15061605

Plus, there are already new IDEs showing up, for example https://chrome.google.com/webstore/detail/kantu-browser-auto...

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#24

Post by hugh7 »

testcross wrote: post #21

Do you know if it is possible to render a page without serving it from a web server? For example, I have the html of one page of my domain generated by a test. I would like to use puppeteer to render it. But I don't want to setup a http server for this. I would like to give a string with the html + a url to page.goto and let it render the page like it comes from the real server. I guess I can cheat by intercepting th…

This is the point of chrome headless.

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#25

Post by hugh7 »

twsted wrote: post #14

Two things: 1. Please do not test a web app with Chrome only, we don't want to go back to a world with a single browser 2. > So, until puppeteer supports this, we will rely on jsdom, a package available via npm JSDOM is not just a package on npm, it's an engineering piece of art

I have given up on Firefox. Every bug I submit takes it own sweet time to get fixed.

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#26

Post by egeozcan »

houli wrote: post #22
testcross wrote: post #21

Do you know if it is possible to render a page without serving it from a web server? For example, I have the html of one page of my domain generated by a test. I would like to use puppeteer to render it. But I don't want to setup a http server for this. I would like to give a string with the html + a url to page.goto and let it render the page like it comes from the real server. I guess I can cheat by intercepting th…

You should be able to use a data URI containing the HTML string

It's not allowed anymore: https://groups.google.com/a/chromium.org/forum/m/#!topic/bli...

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#28

Post by stokilo »

testcross wrote: post #21

Do you know if it is possible to render a page without serving it from a web server? For example, I have the html of one page of my domain generated by a test. I would like to use puppeteer to render it. But I don't want to setup a http server for this. I would like to give a string with the html + a url to page.goto and let it render the page like it comes from the real server. I guess I can cheat by intercepting th…

Save response in a file and open in puppeteer using file protocol smth like file:///c:/response.html

Re: Show HN: Getting started with Puppeteer and Chrome Headless for Web Scraping

#30

Post by paps »

Where I work we prefer jQuery to the native DOM API for scraping. It really speeds up the process of extracting data.

For example with Puppeteer you can do page.injectFile("jquery-3.2.1.min.js"). I think that would simplify your evaluate() calls.

It would also be easy to speed up the whole process by doing a single evaluate() call per page with all your scraping code in it.

BTW we just released an article with tips & tricks for Headless Chrome: https://blog.phantombuster.com/web-scraping-in-2017-headless... What do you think?

Post reply on HN