Live data from Hacker News

PhantomJS: Stepping down as maintainer

groups.google.com

41–50 of 236 posts

Re: PhantomJS: Stepping down as maintainer

#41

I'm curious - What is the utility of headless browsers? Are there people who earn money by getting it to automatically fill out forms, enter competitions etc?

We used it a lot for full automation tests for the UI. It's nice being able to interface with a full-featured browser that can run javascript, etc. And take screenshots when things go wrong.

Re: PhantomJS: Stepping down as maintainer

#42
post #37

Earlier quoted context omitted.

While we're on the topic, does anyone know where one might find scripts to scrape bank statements so you don't have to download them manually every month? (This is one thing I would find headless browsers useful for...)

If your authentication is basic enough, it should be a problem to write one. The problem with a lot of banks is things like two-factor auth to do anything . If I didn't have a mortgage locked in at a crazily great rate, I'd consider changing banks just to get one that'll let me automate statement downloads. The alternative is to build a device to press buttons on their 2FA device... Come to thing of it that might be…

2FA isn't my issue here. Actually downloading the statements is. Lots of banks use JS, some go through really weird hoops getting you the PDF that are difficult for non-experienced people to automate. Stuff like JS in embedded iframes that generate the link on the fly and open a new tab that you have to navigate. It's hard to accurately detect all the links and handle things like "Next Page" and so on, especially for more than one bank. It's quite nontrivial.

Re: PhantomJS: Stepping down as maintainer

#44
post #32

Open source software has to be one of the least efficient markets out there. If you sum up the very real value PhantomJS has delivered to very real companies over the last several years, napkin math tells me we wouldn't have the project being abandoned for being a "bloody hell" to work on.

It's a classic "public good" - an economic activity whose benefits are impractical or unworthwhile to deny to those who don't pay. Things like emergency services, last-mile road infrastructure, and environmental protection work. A special-case of positive externalities. These are a classic example used by economists of a natural place for government in the economy.

Re: PhantomJS: Stepping down as maintainer

#45

Earlier quoted context omitted.

While we're on the topic, does anyone know where one might find scripts to scrape bank statements so you don't have to download them manually every month? (This is one thing I would find headless browsers useful for...)

Every UK bank I've used allows some format of csv/qif/ofx export. Are you in the US?

They generally do. But many UK banks make it hard to automate things. Either insisting on 2FA in all cases, or having a secondary login without 2FA that only gives very limited access.

Some way of authorising API access to read-only access to things like statements would be fantastic, to the extent that I'd consider changing banks over it, if you know of any UK banks that offer it.

Re: PhantomJS: Stepping down as maintainer

#46

I'm curious - What is the utility of headless browsers? Are there people who earn money by getting it to automatically fill out forms, enter competitions etc?

I wrote a PhantomJS script to download data from my bank accounts. They offer no API and disfunctional text-based exports, their websites are ridden with "good" (=terrible) web & security practices, like 3-characters-of-the-whole-password authentication, single-tab sessions, frames, etc. that makes it pretty much impossible to scrape with Python but relatively easy with a fully-fledged browser (although that still requires a lot of bank-specific boilerplate code).

Re: PhantomJS: Stepping down as maintainer

#48

Besides testing, my team uses phantomjs to convert pages to PDF and also to convert javascript generated charts to images. This is sad news, I'm don't think chrome eventually will add support for.

There's wkhtmltopdf[1] when you need the PDF functionality. It is licensed with LGPL, so you can make it interop with your commercial product.

[1]: https://wkhtmltopdf.org/

Re: PhantomJS: Stepping down as maintainer

#49
post #32

Open source software has to be one of the least efficient markets out there. If you sum up the very real value PhantomJS has delivered to very real companies over the last several years, napkin math tells me we wouldn't have the project being abandoned for being a "bloody hell" to work on.

> Open source software has to be one of the least efficient markets out there.

The regular market rules don't really apply to Open source software. A lot of viable (or even thriving) open source projects would be dismal failures as stand-alone businesses or startups. Paradoxically, the only way they can provide real value is in their current form of open source projects.

I would have given CyanogenMod as an example, but the amount of inept management at the startup there would cloud the issue.

Re: PhantomJS: Stepping down as maintainer

#50
post #37

Earlier quoted context omitted.

If your authentication is basic enough, it should be a problem to write one. The problem with a lot of banks is things like two-factor auth to do anything . If I didn't have a mortgage locked in at a crazily great rate, I'd consider changing banks just to get one that'll let me automate statement downloads. The alternative is to build a device to press buttons on their 2FA device... Come to thing of it that might be…

2FA isn't my issue here. Actually downloading the statements is. Lots of banks use JS, some go through really weird hoops getting you the PDF that are difficult for non-experienced people to automate. Stuff like JS in embedded iframes that generate the link on the fly and open a new tab that you have to navigate. It's hard to accurately detect all the links and handle things like "Next Page" and so on, especially for…

If you don't have 2FA issues, then while I agree it's non-trivial, it's certainly doable with a headless browser. But yes, I'd love for there to be simpler ways to do this in general.
Post reply on HN