Import.io, Kimono Labs, and now this. Web scraper -> data area is heating up.
I also wrote http://scrape.ly, which let's you write web scrapers via a URL syntax.
Portia, an open-source visual web scraper
21–30 of 70 posts
Re: Portia, an open-source visual web scraper
#22I just took it for a testdrive and it was an absolute pleasure. I tried to scrape all job listings at https://hasjob.co hoping to find trends.
There is one small pain, the output is being printed to the console and piping output to file is not figuring. But it did fetch all the pages and printed a nice json.
UPDATE: there is a logfile setting to dump output to file
Re: Portia, an open-source visual web scraper
#23Love this, been using Scrapy for all my scraping needs.
Re: Portia, an open-source visual web scraper
#24Re: Portia, an open-source visual web scraper
#25Is there a live demo available?
Not affiliated with the OP or the project, but I threw up a sandbox to play with this at http://awstest123.notesies.com:9001/static/main.html
Re: Portia, an open-source visual web scraper
#26The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...
Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?
Re: Portia, an open-source visual web scraper
#27The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...
Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?
PhantomJS?
Re: Portia, an open-source visual web scraper
#28Earlier quoted context omitted.
Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?
PhantomJS?
"PhantomJS is a headless WebKit scriptable with a JavaScript API", so it's a browser.
Is it also a webscraper that can pull data out of a page for me?
Re: Portia, an open-source visual web scraper
#29Re: Portia, an open-source visual web scraper
#30The problem with these sorts of solutions is that they work perfectly for 'simple' sites like the register, but fail hard with 'modern' sites like, e.g. ASOS.com. Just tried ASOS and the web front end failed to request a product page correctly... All the dynamic JS and whatnot just plays havoc with these projects. In my experience you have to run through webdriver or something like phantomjs and parse the JS...
Anybody know of any tools that would work with JS-rendered sites, and not have to "parse the JS"?
Answering my own question:
CasperJS is an open source navigation scripting & testing utility written in Javascript for the PhantomJS WebKit headless browser and SlimerJS (Gecko). It eases the process of defining a full navigation scenario and provides useful high-level functions, methods & syntactic sugar for doing common tasks such as:
defining & ordering browsing navigation steps
filling & submitting forms
clicking & following links
capturing screenshots of a page (or part of it)
testing remote DOM
logging events
downloading resources, including binary ones
writing functional test suites, saving results as JUnit XML
scraping Web contents