As has been said, this point was somewhat inevitable with the advent of Chrome and Firefox's headless modes. However, as the project slips into the mists of history, let's not forget the vital stepping stone it provided in having access to a real headless browser environment vs a simulated one. I for one will remain grateful to Ariya, Vitallium and all the team for their efforts.
PhantomJS: Archiving the project, suspending development
11–20 of 138 posts
Re: PhantomJS: Archiving the project, suspending development
#12Earlier quoted context omitted.
Maintaining systems already built on top of PhantomJS.
A bit concerning, as youtube-dl relies on PhantomJS currently.
youtube-dl will do fine. It is updated once in several days, and with that activity count, I think they will transition to headless chrome in no time.
Re: PhantomJS: Archiving the project, suspending development
#13Well with Chrome going headless there isn't a whole lot of place for PhantomJS anyway. Or is there? What is it still good for?
Legacy systems for one. The Cooperative Patent Classification group releases their classifications en masse as HTML (single zip download, which is great). I built a parser for a PHP project that could parse all several hundred thousand records from the HTML in a few minutes. In 2017, they switched to a system that loads in the data from JSON stored in Javascript in the HTML (it is every bit as terrible as you imagine…
Why would you need PhantomJS for that? Can't you just parse the HTML files with Nokogiri and be done with it? That would be orders of magnitude faster anyway
Re: PhantomJS: Archiving the project, suspending development
#14Well with Chrome going headless there isn't a whole lot of place for PhantomJS anyway. Or is there? What is it still good for?
Not sure whether it is as easy to use as PhantomJS.
Well that depends if you're stuck with Javascript. There isn't anything simpler (that I'm aware of - bu I do web scraping/automation professionally for about 6 years) than watir[0]. PhantomJS doesn't even come remotely close.
Re: PhantomJS: Archiving the project, suspending development
#15Earlier quoted context omitted.
Legacy systems for one. The Cooperative Patent Classification group releases their classifications en masse as HTML (single zip download, which is great). I built a parser for a PHP project that could parse all several hundred thousand records from the HTML in a few minutes. In 2017, they switched to a system that loads in the data from JSON stored in Javascript in the HTML (it is every bit as terrible as you imagine…
Why would you need PhantomJS for that? Can't you just parse the HTML files with Nokogiri and be done with it? That would be orders of magnitude faster anyway
I had to actually render the HTML and run the Javascript in order to populate the HTML with the data I needed to parse. The HTML does not include the parse-able data by default and is populated at runtime from JSON embedded in the Javascript in the HTML.
As far as I am aware, Nokogiri isn't capable of that and even if it is, I was unaware of that library at the time I wrote the Phantom solution (only discovered it last Summer but have yet to use it for anything)
Re: PhantomJS: Archiving the project, suspending development
#16Chrome and Firefox gaining headless modes is the ultimate goal Phantom could've achieved. So I consider it a complete success. Kudos to all contributors.
I feel the same. No failure here. It served its purpose very well, while it was needed. It’s retiring while it’s still better than it’s replacement in a few areas...like proxy support.
Re: PhantomJS: Archiving the project, suspending development
#17Some people are mentioning headless Chromium, so I wanna mention another tool I've used to replace some of phantomjs' functionality: jsdom [0].
It's much more lightweight than a real browser, and it doesn't require large extra binaries.
I don't do any complex scrapping, but occasionally I want to pull down and aggregate a site's data. For most pages, it's as simple as making a request and passing the response into a new jsdom instance. You can then query the DOM using the same built-in browser APIs you're already familiar with.
I've previously used jsdom to run a large web app's tests on node, which provided a huge performance boost and drastically lowered our build times. As long as you maintain a good architecture (i.e. isolating browser specific bits from your business logic) you're unlikely to encounter any pitfalls. Our testing strategy was to use node and jsdom during local testing and on each commit. IMO, you should generally only need to run tests on an actual browser before each release (as a safety net), and possibly on a regular schedule (if your release cycle is long).
Re: PhantomJS: Archiving the project, suspending development
#18Earlier quoted context omitted.
Why would you need PhantomJS for that? Can't you just parse the HTML files with Nokogiri and be done with it? That would be orders of magnitude faster anyway
I had to actually render the HTML and run the Javascript in order to populate the HTML with the data I needed to parse. The HTML does not include the parse-able data by default and is populated at runtime from JSON embedded in the Javascript in the HTML. As far as I am aware, Nokogiri isn't capable of that and even if it is, I was unaware of that library at the time I wrote the Phantom solution (only discovered it la…
No, Nokogiri isn't capable of that so you need an actual browser runtime. I didn't think a downloadable site would have javascript populating the page with data. But if it's only from JSON embedded in the JS from the HTML then I guess it's still possible to retrieve that and unless it requires some processing a JSON is as good as you can get.
Re: PhantomJS: Archiving the project, suspending development
#19Chrome and Firefox gaining headless modes is the ultimate goal Phantom could've achieved. So I consider it a complete success. Kudos to all contributors.
Agree 100%. Headless & programmable browsing is a hard nut to crack. PhantomJs paved the way.
Re: PhantomJS: Archiving the project, suspending development
#20Chrome and Firefox gaining headless modes is the ultimate goal Phantom could've achieved. So I consider it a complete success. Kudos to all contributors.
I really hope Microsoft offers a headless IE / Edge at some point. It would be amazing to be able to use all 3 major browsers like this. Heck, get Safari in there too (though I feel like it should be doable with WebKit already).