21–29 of 29 posts
Great work! I really like this! I typically use the JavaScript console bookmarklet for tasks like this, but it is not specifically designed for scraping. I would love to see an option that would allow Artoo commands to be packaged into a PhantomJS script. Developers could use Artoo manually to figure out what elements should be targeted and then the PhantomJS script to run it in an automated fashion.
I don't know if it is possible, but could this run as a Chrome Extension, in a background script, loading various pages, executing code on then and keep going, storing the data at the extension's localStorage?
It could also store the code of the scrapers, for reusing.
This is awesome. I've been dreaming about this for weeks. I don't know if it is possible, but could this run as a Chrome Extension, in a background script, loading various pages, executing code on then and keep going, storing the data at the extension's localStorage? It could also store the code of the scrapers, for reusing.
This is awesome. I've been dreaming about this for weeks. I don't know if it is possible, but could this run as a Chrome Extension, in a background script, loading various pages, executing code on then and keep going, storing the data at the extension's localStorage? It could also store the code of the scrapers, for reusing.
Well, I see you already have almost all I suggested. Now I would want something to make the ajaxSpider render the pages using the browser engine, instead of just getting pure HTML.
I would like to see something that helps create useful, specific scrapers for languages like Ruby and Python. It's annoying to have to run scripts multiple times, tweaking it after each run to get exactly what you need. It's a waste of time...
>> ipython
[In 1]: from pyquery import PyQuery as pq
[In 2]: pq("http://www.foo.com")("")
... or do it with requests + lxml.etree, or whatever you want
when you have what you need, copy and paste into a file
Might be better to use another name?
What advantages does this have over Phantom.js[1] ? [1] http://phantomjs.org
But combining both would be nice to make it possible to automatize scrapers that have been developed quickly directly in the browser with artoo.
I would like to see something that helps create useful, specific scrapers for languages like Ruby and Python. It's annoying to have to run scripts multiple times, tweaking it after each run to get exactly what you need. It's a waste of time...
>> ipython [In 1]: from pyquery import PyQuery as pq [In 2]: pq("http://www.foo.com")(" ") (inspect output, repeat till right) ... or do it with requests + lxml.etree, or whatever you want when you have what you need, copy and paste into a file