Live data from Hacker News

Puppeteer: Headless Chrome Node API

github.com

91–100 of 105 posts

Re: Puppeteer: Headless Chrome Node API

#91
post #52
post #47

Does anyone know if it's possible to take a screenshot of a specific DOM element? (instead of the entire page)

Currently only PDF printing of the entire page view-port is available. File an issue with your use-case so the team can assess whether or not to look into including that kind of behavior.

It is already supported, as Paul Irish said

Re: Puppeteer: Headless Chrome Node API

#92
post #10
post #9

So first there was Selenium's JSON Wire protocol, then came the W3C WebDriver spec and now we're back to browser-specific implementations? As someone who's tried/is trying to automate Firefox/Chrome/Safari/IE in a consistent fashion, my only question is: WHY?

Because WebDriver just doesn't do. For an example of an impossible task, try to retrieve request headers using Selenium. Browser automation getting more and more complicated the more cases you're trying to cover and in my impression WebDriver is definitely not enough. Who knows, perhaps some new version of WebDriver that I never heard of it will catch up once the functionality gets properly defined.

For an example of an impossible task, try to retrieve request headers using Selenium.

Selenium can't do that itself but Selenium can drive a browser that uses a proxy and you can retrieve everything about the request from that. It's a lot more challenging if you're testing things that use SSL but it's not impossible with a decent proxy app (eg Charles).

Re: Puppeteer: Headless Chrome Node API

#93

Earlier quoted context omitted.

Chrome DevTools Protocol (CDP) is a more advanced API, and I've been leading an effort on getting multiple browsers to look at CDP under https://remotedebug.org

Get them to actually version their protocol first, maybe? As someone with a bunch of tooling around the dev-tools API, it's a huge pain in the ass to not be able to tell what functions the remote browser supports. There's a version number in the protocol description, but it's literally never been incremented as far as I've seen. https://github.com/ChromeDevTools/devtools-protocol/issues/6

File a bug for this.

It should be possible to get the running chrome version and to do feature detection over this protocol.

If it isn't, it's a bug

Re: Puppeteer: Headless Chrome Node API

#95
post #63

This is great, but it's sobering to see how hard it is to get a nice, "complete" PDF screenshot out of a modern site. Here's a quick hack: https://gist.github.com/rcarmo/cf698b52832d0ec356c147cf9c9ad... I'm using The Verge for testing because it lazy loads images, and am being clumsy about the scrolling, but it mostly works - I can get 90% of the images to show on the finished PDF. What I can't seem to get right, tho…

If I'm reading [1] right, you want to use getBoundingClientRect() instead of clientHeight, which does not account for borders and margins.

1: https://developer.mozilla.org/en-US/docs/Web/API/CSS_Object_...

Re: Puppeteer: Headless Chrome Node API

#96
post #63

This is great, but it's sobering to see how hard it is to get a nice, "complete" PDF screenshot out of a modern site. Here's a quick hack: https://gist.github.com/rcarmo/cf698b52832d0ec356c147cf9c9ad... I'm using The Verge for testing because it lazy loads images, and am being clumsy about the scrolling, but it mostly works - I can get 90% of the images to show on the finished PDF. What I can't seem to get right, tho…

If I'm reading [1] right, you want to use getBoundingClientRect() instead of clientHeight, which does not account for borders and margins. 1: https://developer.mozilla.org/en-US/docs/Web/API/CSS_Object_...

Well, no, really. Seems like this is due to the way Chrome deals with print CSS and PDF generation rather than any kind of calculations I do prior to invoking PDF generation...

Re: Puppeteer: Headless Chrome Node API

#97
post #12

One of the biggest wins here is this little tidbit: > When you install Puppeteer, it downloads a recent version of Chromium (~71Mb Mac, ~90Mb Linux, ~110Mb Win) that is guaranteed to work with the API. A lot of the chrome interface libs about at the moment require you to maintain your own instance of chrome/chromium and launch the headless server with your command line, or require a pre compiled version, that can qui…

Just grab the most recent Chrome/Chromium .deb from the dev/canary/stable channel via a cron job. Piece of cake. We do it for our services.

That said, headless still has a mountain of bugs open against it, so most parties will be best suited by sticking with ChromeDriver for now.

Re: Puppeteer: Headless Chrome Node API

#98

Earlier quoted context omitted.

Get them to actually version their protocol first, maybe? As someone with a bunch of tooling around the dev-tools API, it's a huge pain in the ass to not be able to tell what functions the remote browser supports. There's a version number in the protocol description, but it's literally never been incremented as far as I've seen. https://github.com/ChromeDevTools/devtools-protocol/issues/6

File a bug for this. It should be possible to get the running chrome version and to do feature detection over this protocol. If it isn't, it's a bug

Did.... did you notice the link to the bug report in my comment?

Re: Puppeteer: Headless Chrome Node API

#99
post #83

Earlier quoted context omitted.

I think this is more "selenium is actively antagonistic to it's major use-case", then trying to throw everything away. There have been multiple attempts to convince the selenium people to revisit their decision W.R.T. headers, and they're completely unwilling. Given that the selenium leadership is apparently uninterested in improvements, and it's many limitations, trying to improve there is more effort then it's wort…

I didn't follow that development. Can you share why Selenium maintainers chose not to implement headers? Is it that they want to restrict the tool to simulate what a normal user can do with a browser and not hacks such as overriding headers? Thanks in advance!

"Something something something normal users can't do that something something".

Basically, they're still stuck in this idea that they're ONLY for emulating user-available input (and the dev-tools don't exist).

In reality, there is tremendous interest in more complex programmatic interfaces, but apparently they're unwilling to see that, and are instead only interested in their implementations "user-only" ideological purity.

Re: Puppeteer: Headless Chrome Node API

#100

Earlier quoted context omitted.

I do a lot of web scraping that requires a full browser for some websites. This sounds perfect for me. I often use Selenium, but it's a lot of complexity and quite buggy if I just want to run chrome (or any one browser, but not all browsers).

Have you tried TestCafe? I have had good experience with TestCafe on a recent project. The development team are very responsive to bug fixing and stability.

Hadn't heard of it before, I have an upcoming project that's going to need some JS scraping, so I'll give this a shot as well as Puppeteer. Thanks!
Post reply on HN