Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Portia, an open-source visual web scraper

blog.scrapinghub.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: Portia, an open-source visual web scraper

#31

Post by e1g »

checker659 wrote: post #27
egb wrote: post #26

Earlier quoted context omitted.

Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?

PhantomJS?

>Is it also a webscraper that can pull data out of a page for me?

No, Phantom will only recreate the page as it would look like to a human user (i.e. after all javascript is parsed and executed). It will not help you parse or slice the page - you would have to do that part programatically with other dom-parsing tools.

Re: Portia, an open-source visual web scraper

#32

Post by nols »

egb wrote: post #28
checker659 wrote: post #27

Earlier quoted context omitted.

PhantomJS?

"PhantomJS is a headless WebKit scriptable with a JavaScript API", so it's a browser. Is it also a webscraper that can pull data out of a page for me?

I've heard of people using PhantomJS with CasperJS to scrape, not sure if it can be done solely with PhantomJS.

Re: Portia, an open-source visual web scraper

#34

Post by checker659 »

e1g wrote: post #31
checker659 wrote: post #27

Earlier quoted context omitted.

PhantomJS?

>Is it also a webscraper that can pull data out of a page for me? No, Phantom will only recreate the page as it would look like to a human user (i.e. after all javascript is parsed and executed). It will not help you parse or slice the page - you would have to do that part programatically with other dom-parsing tools.

Sorry. I guess you're right. As a programmer, both of those look the same to me.

Re: Portia, an open-source visual web scraper

#35

Post by alttab »

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue.

In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need.

To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, that you could place inside any page and utilize local storage, you could scrape these dynamic sites by viewing the page first, and still get all of the cool gadetry provided by this tool.

In essence, the value of this tool could be built as a bookmarklet. THAT SIR - I would use every, single, day.

Re: Portia, an open-source visual web scraper

#36

Post by CHsurfer »

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

At first, this seems correct. It's definately easier to get scraping with something like Capybara and a suitable js enabled driver, but in my experience, this solution is less reliable. Async loaded data can time out and don't get me started on the difficulties of running the scraper with cron jobs. In the end, I migrated even my JS heavy pages to Mechanize based solutions. It takes a few extra requests to get the async data, but once you get that figured out, it's rock solid - till they update the site design ;-)

Re: Portia, an open-source visual web scraper

#37

Post by CHsurfer »

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

At first, this seems correct. It's definately easier to get scraping with something like Capybara and a suitable js enabled driver, but in my experience, this solution is less reliable. Async loaded data can time out and don't get me started on the difficulties of running the scraper with cron jobs. In the end, I migrated even my JS heavy pages to Mechanize based solutions. It takes a few extra requests to get the async data, but once you get that figured out, it's rock solid - till they update the site design ;-)

Re: Portia, an open-source visual web scraper

#38

Post by beernutz »

I really dig these scrapers, but most of them seem to only work well for simple sites as someone has already noted.

Just want to point out a (commercial but reasonable) program that really works well for all our odd edge case customer site issues.

http://www.visualwebripper.com

Re: Portia, an open-source visual web scraper

#40

Post by vaviloff »

nols wrote: post #32
egb wrote: post #28

Earlier quoted context omitted.

"PhantomJS is a headless WebKit scriptable with a JavaScript API", so it's a browser. Is it also a webscraper that can pull data out of a page for me?

I've heard of people using PhantomJS with CasperJS to scrape, not sure if it can be done solely with PhantomJS.

CasperJS is a higher-level wrapper for PhantomJS, so - yes, it could be done with PhantomJS solely... But you wouldn't want to, because CasperJS makes automation easier.
Post reply on HN