Portia, an open-source visual web scraper

Post by lifeisstillgood » Wed, Apr 02, 2014, 4:57 AM UTC

kh_hk wrote: post #57

lifeisstillgood wrote: post #56

Earlier quoted context omitted.

Surely there are difficulties in expecting data providers to produce their data in standard formats across industries and countries? I am naive as to how much and what data is available but that seems a stretch

If interested, take a look at my project on unifying bike sharing networks data. Besides providing a public API, we are also providing a python library that accesses and abstracts different sources under the same model [1, 2] There are a lot of accessible sources (though, not documented), but there are also clear examples on how one would never provide a service! Some examples [3, 4] What I was referring, though, was…

Post by umurkontaci » Wed, Apr 02, 2014, 9:21 AM UTC

notastartup wrote: post #60

alttab wrote: post #35

Earlier quoted context omitted.

There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, tha…

how would a bookmarklet be able to crawl & scrape a website?

Post by notastartup » Wed, Apr 02, 2014, 3:05 PM UTC

shaneofalltrad wrote: post #59

alttab wrote: post #35

Earlier quoted context omitted.

There are multiple internal tools I use at work (JIRA, our ticketing system, our code review tool) that won't work because of this issue. In the meantime, I've written Tampermonkey scripts that will scrape and embedd multiple pages all hack-like, but at least I get a good CSV of the data I need. To me, the value in this tool is the user interface for creating the scrape logic. If this ran as an embeddable JS app, tha…

Great idea on the bookmarklet. I could see a tool for building custom readers with clippings from various sites. Say I want to organize JavaScript array patterns and ideas. Throw in a way to clip parts of my PDF books into this "reader" and you have an amazing product worth millions.

Post by CMCDragonkai » Thu, Apr 03, 2014, 6:03 AM UTC

egb wrote: post #26

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...

Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?

Post by CMCDragonkai » Thu, Apr 03, 2014, 6:08 AM UTC

dabeeeenster wrote: post #11

The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...