Ariya wrote a bit about his reasoning here: https://mobile.twitter.com/AriyaHidayat/status/9701730017013...
also mentioning an old post in https://github.com/ariya/phantomjs/issues/14541
PhantomJS: Archiving the project, suspending development
51–60 of 138 posts
Re: PhantomJS: Archiving the project, suspending development
#52Earlier quoted context omitted.
I really hope Microsoft offers a headless IE / Edge at some point. It would be amazing to be able to use all 3 major browsers like this. Heck, get Safari in there too (though I feel like it should be doable with WebKit already).
they have webdriver support. Is that enough? I'm not sure what's the difference between headless and webdriver (if any) https://blogs.windows.com/msedgedev/2015/07/23/bringing-auto...
Running Edge with webdriver still includes the GUI.
Re: PhantomJS: Archiving the project, suspending development
#53One of the guys working on P-JS just linked from a GH issue to his open letter... He isn't very happy with the owner blah blah blah and is going to fork the master branch to make phantom great again, I'll just put this here:
"Will do as advised, as I really think PhantomJS is good project, it just needs good, devoted leader."
Re: PhantomJS: Archiving the project, suspending development
#54Well with Chrome going headless there isn't a whole lot of place for PhantomJS anyway. Or is there? What is it still good for?
Legacy systems for one. The Cooperative Patent Classification group releases their classifications en masse as HTML (single zip download, which is great). I built a parser for a PHP project that could parse all several hundred thousand records from the HTML in a few minutes. In 2017, they switched to a system that loads in the data from JSON stored in Javascript in the HTML (it is every bit as terrible as you imagine…
Is this the data you were trying to parse?
https://www.cooperativepatentclassification.org/Archive.html
Re: PhantomJS: Archiving the project, suspending development
#55Earlier quoted context omitted.
Legacy systems for one. The Cooperative Patent Classification group releases their classifications en masse as HTML (single zip download, which is great). I built a parser for a PHP project that could parse all several hundred thousand records from the HTML in a few minutes. In 2017, they switched to a system that loads in the data from JSON stored in Javascript in the HTML (it is every bit as terrible as you imagine…
Why would you need PhantomJS for that? Can't you just parse the HTML files with Nokogiri and be done with it? That would be orders of magnitude faster anyway
I think you might have missed this part:
>In 2017, they switched to a system that loads in the data from JSON stored in Javascript in the HTML
Re: PhantomJS: Archiving the project, suspending development
#56There is one thing about this that saddens me: PhantomJS still starts up much faster than headless Firefox or Chrome, at least for me, which makes some of our integration tests take a long longer than they should. Has anyone here figured out any tricks to get headless Chrome booted fast?
Also PhantomJS was a single statically linked binary with no dependencies that you could literally drop into a server and run scripts at once.
For those who may struggle with using chrome headless on server, here is a dockerfile example to get your started : https://github.com/oelmekki/chromessr/blob/master/Dockerfile
godet is the lib I use for chrome piloting, replace with your favorite one.
Re: PhantomJS: Archiving the project, suspending development
#57I do lightweight web automation via Chromiums "Snippets". It is super nice to work that way because you see on screen what happens and can check everything realtime in the console. Only problem is that they dont survive page loads. So when my snippet navigates to a new url I have to trigger it again manually. What would be a good way to progress from here so I can automate across pages?
Re: PhantomJS: Archiving the project, suspending development
#58Earlier quoted context omitted.
OK, obviously the stuff you see on your screen not matching the HTML delivered makes sense, but explain the HTML source not matching what's sent via the HTTP response. DOM can be modified, of course, JS can introduce more dynamic HTML, but view-source should always represent any non-redirected HTTP response. What is Nokogiri getting that the browser isn't (or vice versa)?
> JS can introduce more dynamic HTML, but view-source should always represent any non-redirected HTTP response That is both true and false. Because the JS can introduce dynamic content, the source returned by the HTTP response often doesn't match the source that is rendered by the browser itself. In many cases, a site will return a skeleton (just HTML) and then make an Ajax request to populate it. In my case, it was…
But we're not talking about the rendered source here. We're talking about "view source", which afaik always matches what is returned by the server.
The post replied to claims that Nokogiri doesn't see this however so I'm puzzled.
Re: PhantomJS: Archiving the project, suspending development
#59Earlier quoted context omitted.
OK, obviously the stuff you see on your screen not matching the HTML delivered makes sense, but explain the HTML source not matching what's sent via the HTTP response. DOM can be modified, of course, JS can introduce more dynamic HTML, but view-source should always represent any non-redirected HTTP response. What is Nokogiri getting that the browser isn't (or vice versa)?
> view-source should always represent any non-redirected HTTP response Not the grandfather, but generally in browsers you have two versions of HTML "source" - the canonical source, the stuff pulled down over HTTP, and the repaired source, the version that actually gets rendered. I'm unfamiliar with Nokogiri, but I suspect that from context, it doesn't repair HTML in the same way that browsers do.
But it should be the same as "view source" right? The post replied to claims otherwise.
Re: PhantomJS: Archiving the project, suspending development
#60Thank you, PhantomJS contributors. You built a life saver.