Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PhantomJS: Archiving the project, suspending development

github.com

Post reply on HN
Print view View original

71–80 of 138 posts

Re: PhantomJS: Archiving the project, suspending development

#71

Post by dewey »

Kiro wrote: post #58
apocalyptic0n3 wrote: post #31

Earlier quoted context omitted.

> JS can introduce more dynamic HTML, but view-source should always represent any non-redirected HTTP response That is both true and false. Because the JS can introduce dynamic content, the source returned by the HTTP response often doesn't match the source that is rendered by the browser itself. In many cases, a site will return a skeleton (just HTML) and then make an Ajax request to populate it. In my case, it was…

But we're not talking about the rendered source here. We're talking about "view source", which afaik always matches what is returned by the server. The post replied to claims that Nokogiri doesn't see this however so I'm puzzled.

"view source" shows the source after all the javascript ran. So what a client that doesn't execute javascript (like curl) sees is different from what you see in "view source".

That's also the reason while you had to "pre-render" you javascript web apps for SEO purposes until google bot got the ability to execute javascript.

Re: PhantomJS: Archiving the project, suspending development

#72

Post by dewey »

Kiro wrote: post #59
joatmon-snoo wrote: post #27

Earlier quoted context omitted.

> view-source should always represent any non-redirected HTTP response Not the grandfather, but generally in browsers you have two versions of HTML "source" - the canonical source, the stuff pulled down over HTTP, and the repaired source, the version that actually gets rendered. I'm unfamiliar with Nokogiri, but I suspect that from context, it doesn't repair HTML in the same way that browsers do.

But it should be the same as "view source" right? The post replied to claims otherwise.

No it's not. https://news.ycombinator.com/item?id=16514517

Re: PhantomJS: Archiving the project, suspending development

#73

Post by madeofpalk »

TheAceOfHearts wrote: post #50
AlphaWeaver wrote: post #29

Earlier quoted context omitted.

Cheerio [0] is fantastic for this as well... [0]: https://www.npmjs.com/package/cheerio

I've tried Cheerio as well, but I prefer JSDOM since it exposes the DOM APIs. What I'll normally do is interactively test things out in the browser's console, and then transfer em over to my script. Browser dev tools are just super amazing.

Agreed - I find the Cheerio APIs to be awkward when traversing deep into the DOM. Last time I used Beautiful Soup I found it also had this problem. The DOM API that JSDOM provides is such much more natural to work with.

Re: PhantomJS: Archiving the project, suspending development

#74

Post by madeofpalk »

TheAceOfHearts wrote: post #17

Some people are mentioning headless Chromium, so I wanna mention another tool I've used to replace some of phantomjs' functionality: jsdom [0]. It's much more lightweight than a real browser, and it doesn't require large extra binaries. I don't do any complex scrapping, but occasionally I want to pull down and aggregate a site's data. For most pages, it's as simple as making a request and passing the response into a…

One question I've had recently is how to scrape out a Javascript object out of HTML source. With server-side react + redux, I've wanted to be able to scrap out the serialised var __STATE__ = {...} object to JSON, from nodejs. Best solution I cobbled together was to basically eval() the JS source, which I know is far from ideal.

Re: PhantomJS: Archiving the project, suspending development

#75

Post by madeofpalk »

dewey wrote: post #71
Kiro wrote: post #58

Earlier quoted context omitted.

But we're not talking about the rendered source here. We're talking about "view source", which afaik always matches what is returned by the server. The post replied to claims that Nokogiri doesn't see this however so I'm puzzled.

"view source" shows the source after all the javascript ran. So what a client that doesn't execute javascript (like curl) sees is different from what you see in "view source". That's also the reason while you had to "pre-render" you javascript web apps for SEO purposes until google bot got the ability to execute javascript.

I get what you're saying now, but I believe you're mistaken about "View Source".

I've never seen "View Page Source" or "Show Page Source" be the current DOM representation. It's always the HTML what came over the wire, the same you'll get from curl (unless the server is going user agent shenanigans, which I think we can agree is out of scope here).

If you're talking about the page after Javascript is ran, the only way you're seeing that is by opening the dev tools and looking in the 'Elements' or 'Inspector' panel.

I just checked in Safari, Chrome, and Firefox and found this to be true in all of them. The distinction between the View Source and DOM Inspector is very clear.

Re: PhantomJS: Archiving the project, suspending development

#76

Post by detaro »

dewey wrote: post #71
Kiro wrote: post #58

Earlier quoted context omitted.

But we're not talking about the rendered source here. We're talking about "view source", which afaik always matches what is returned by the server. The post replied to claims that Nokogiri doesn't see this however so I'm puzzled.

"view source" shows the source after all the javascript ran. So what a client that doesn't execute javascript (like curl) sees is different from what you see in "view source". That's also the reason while you had to "pre-render" you javascript web apps for SEO purposes until google bot got the ability to execute javascript.

In what browser is this case? Chrome and Firefox it isn't. In the dev tools, you see the rendered DOM, but view source shows you the HTML from the server.

Re: PhantomJS: Archiving the project, suspending development

#77

Post by poizan42 »

BinaryIdiot wrote: post #65
poizan42 wrote: post #46

Earlier quoted context omitted.

Huh? They have had the WebBrowser ActiveX control since IE3 I think. That works as headless as you get on Windows.

Not quite. It's my understanding it's not as up to date or equivalent to the latest IE. When I tried looking around I just got a lot of confusing information about the control, how it applies (or doesn't) traditional IE hacks, etc. Though I could be wrong but it didn't seem like an equivalent to headless chrome or firefox.

That is not correct, it just runs in compatibility mode by default, see https://blogs.msdn.microsoft.com/patricka/2015/01/12/control...

Re: PhantomJS: Archiving the project, suspending development

#78

Post by acdha »

dewey wrote: post #72
Kiro wrote: post #59

Earlier quoted context omitted.

But it should be the same as "view source" right? The post replied to claims otherwise.

No it's not. https://news.ycombinator.com/item?id=16514517

It sounds like you are confusing View Source and the live developer tools DOM view.

Re: PhantomJS: Archiving the project, suspending development

#79

Post by tokenizerrr »

tonto wrote: post #32
TheAceOfHearts wrote: post #17

Some people are mentioning headless Chromium, so I wanna mention another tool I've used to replace some of phantomjs' functionality: jsdom [0]. It's much more lightweight than a real browser, and it doesn't require large extra binaries. I don't do any complex scrapping, but occasionally I want to pull down and aggregate a site's data. For most pages, it's as simple as making a request and passing the response into a…

Right before release is a bad time to realize there are problems with your build

Depends on how often you release and who sets the schedule.

Re: PhantomJS: Archiving the project, suspending development

#80

Post by seeekr »

madeofpalk wrote: post #74
TheAceOfHearts wrote: post #17

Some people are mentioning headless Chromium, so I wanna mention another tool I've used to replace some of phantomjs' functionality: jsdom [0]. It's much more lightweight than a real browser, and it doesn't require large extra binaries. I don't do any complex scrapping, but occasionally I want to pull down and aggregate a site's data. For most pages, it's as simple as making a request and passing the response into a…

One question I've had recently is how to scrape out a Javascript object out of HTML source. With server-side react + redux, I've wanted to be able to scrap out the serialised var __STATE__ = {...} object to JSON, from nodejs. Best solution I cobbled together was to basically eval() the JS source, which I know is far from ideal.

You could use a parser like esprima or its equivalent from the babeljs ecosystem on the JS source instead and just find the global variable with name `__STATE__` and just eval its init expression. Cheaper, more secure, more direct than actually running the JS.
Post reply on HN