Live data from Hacker News

PhantomJS: Stepping down as maintainer

groups.google.com

161–170 of 236 posts

Re: PhantomJS: Stepping down as maintainer

#161
post #153

Earlier quoted context omitted.

There's wkhtmltopdf[1] when you need the PDF functionality. It is licensed with LGPL, so you can make it interop with your commercial product. [1]: https://wkhtmltopdf.org/

While it's much better than PhantomJS, still causes a lot of issues. We run a paid pet project [1] for html to pdf conversion and most of our customers have stories beginning with PhantomJS then moving to wkhtmltopdf then finally going for something else due to issues with both. Headless Chrome might solve this issue once and for all though. [1] https://restpack.io/html2pdf

[deleted]

Re: PhantomJS: Stepping down as maintainer

#162
post #85

Question for those of you more involved with such headless tasks. Do you think that chromium and firefox supporting headless will induce a surge in bots crawling the open web from now on?

Right off the bat: No. The reason for that is that crawling using a proper browser (i.e. Chrome) is a lot more resource-intensive than using a dedicated tool which only gets the top resource and maybe tries to parse some additional resources. With these kinds of tools you're limited by available bandwidth and IO speeds if you want to store things. If you're looking at a browser, you'll be limited more by things like memory consumption and CPU time, so you'd need a bigger box or more of them to drive the same amount of traffic. There is also not the same amount of ready made applications which take care of crawling, storing and maybe even indexing your data, so not something you can do without actually implementing a lot of things yourself.

Of course, that is only talking about wide-scale scanning. If you're only looking to scrape a single target, for whatever reason, then having an instrumented headless browser will greatly simplify things. Headless Chrome should be more efficient than running it in a (virtual) framebuffer. Plus the whole setup for a powerful crawler is reduced to "install Chrome, start it, point $crawler at the API endpoint". My guess is that we might see turnkey crawling / automation tools appear where you supply a list of URLs and the library + Chrome does the rest. Then, browser-based large scale scanning will be within everyone's reach, only limited by their resources.

Background: I created https://urlscan.io which will simply visit a website and record HTTP interactions (annotating the resources with some helpful meta data). I've been preaching the power of headless instrumented browsers for the better part of a year now ;)

Re: PhantomJS: Stepping down as maintainer

#163

Earlier quoted context omitted.

I think that in the US, we've been fighting for some long about how much to cut government, the idea of proposing to create a new category of spending just doesn't occur to Democrats. The closest they can come to imagining something new is free college (which is the same what we have now [free high school, college scholarships] but more so) and free daycare (which is like free preschool but younger).

You're kind of in a bubble, there is a lot of U.S. government support for open source and most agencies use it. See https://code.gov for a starting point. There's never going to be a huge multi-billion dollar grant program because it is unfair to closed-source companies (who pay taxes) but you are imagining a debate or hangup that doesn't exist.

Why is it unfair? Those companies presumably charge for their software, and maybe will be incentivised to go open source.

Re: PhantomJS: Stepping down as maintainer

#164
post #108
post #79

This is sad, phantomjs is better stripped than Chromium headless, if you ever try to install Chromium on servers without X, it requires shit ton of dependencies, while phantomjs was properly modified requires only minimal library.

I don't really see the problem. After all, you just install those deps once in your Dockerfile, right? ;)

chromium-browser requires these

    chromium-browser chromium-browser-l10n chromium-codecs-ffmpeg-extra cpp
    cpp-4.8 fontconfig fontconfig-config fonts-dejavu-core hicolor-icon-theme
    libasound2 libasound2-data libatk1.0-0 libatk1.0-data libatomic1
    libavahi-client3 libavahi-common-data libavahi-common3 libcairo2
    libcloog-isl4 libcups2 libdatrie1 libdrm-intel1 libdrm-nouveau2
    libdrm-radeon1 libfile-basedir-perl libfile-desktopentry-perl
    libfile-mimeinfo-perl libfontconfig1 libfontenc1 libgdk-pixbuf2.0-0
    libgdk-pixbuf2.0-common libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa
    libgmp10 libgnome-keyring-common libgnome-keyring0 libgraphite2-3
    libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libharfbuzz0b libice6 libisl10
    libjasper1 libjbig0 libjpeg-turbo8 libjpeg8 libllvm3.4 libmpc3 libmpfr4
    libnspr4 libnss3 libnss3-nssdb libpango-1.0-0 libpangocairo-1.0-0
    libpangoft2-1.0-0 libpciaccess0 libpixman-1-0 libsm6 libspeechd2
    libthai-data libthai0 libtiff5 libtxc-dxtn-s2tc0 libx11-xcb1 libxaw7
    libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0
    libxcb-shape0 libxcb-shm0 libxcb-sync1 libxcomposite1 libxcursor1
    libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxmu6 libxpm4
    libxrandr2 libxrender1 libxshmfence1 libxss1 libxt6 libxtst6 libxv1
    libxxf86dga1 libxxf86vm1 x11-common x11-utils x11-xserver-utils xdg-utils

phantomjs:

    fontconfig-config fonts-dejavu-core libfontconfig1 libjpeg-turbo8 libjpeg8

Re: PhantomJS: Stepping down as maintainer

#165
post #150
post #76

Earlier quoted context omitted.

I saw a trick where you can run Chrome and give it less memory, and it uses less memory. This is done using cgroups. The blog post is somewhat old[1] and not in sync with the version that is in Git[2], you might find a way to do this without Docker (I was using an old version of Docker and kernel that couldn't get it to work. But I need the old version of Docker for reasons.) Chrome will aggressively consume any memo…

You can just use cgroups on their own without docker

I would guess ulimit -v might work as well, so you may not even need cgroups.

Re: PhantomJS: Stepping down as maintainer

#167

But can he invert a binary tree? https://twitter.com/mxcl/status/608682016205344768 Edit: some people don't get sarcasm

Yeah, I had the same thought. Just because you have a ton of experience that suggests you'd be good at doing a job doesn't mean a tech company would do something silly like hire you for that job.

Re: PhantomJS: Stepping down as maintainer

#168
post #151

Earlier quoted context omitted.

Who do you bank with?

NatWest, Halifax and AMEX (not a bank, but I want my account data from there as well).

I can't help you with Halifax or AMEX (yet), but my company (https://teller.io/) has a Natwest API in production (private beta). If you would like access, please ping me. sg -at- teller.io

Re: PhantomJS: Stepping down as maintainer

#170

But can he invert a binary tree? https://twitter.com/mxcl/status/608682016205344768 Edit: some people don't get sarcasm

Yeah, I had the same thought. Just because you have a ton of experience that suggests you'd be good at doing a job doesn't mean a tech company would do something silly like hire you for that job.

Furthermore, you may simply be uninterested in the sort of tasks google might give you.
Post reply on HN