Upton: A Web Scraping Framework • Hacker News Board

Re: Upton: A Web Scraping Framework

#61

Post by jeremybmerrill » Tue, Jul 23, 2013, 2:35 PM UTC

geekymartian wrote: post #58

Nice lib, you may take some ideas from pismo: https://github.com/peterc/pismo , more metadata oriented but returns a nokogiri doc as well.

Thanks, will check it out.

Re: Upton: A Web Scraping Framework

#62

Post by Ecio78 » Tue, Jul 23, 2013, 3:19 PM UTC

miket wrote: post #45

Instead of (or in addition to) manually creating scrapers, you can use Diffbot to automatically extract this type of information from news articles using computer vision: http://diffbot.com/products/automatic/article . It also allows you to create rules with a WYSIWYG editor: http://diffbot.com/products/custom/

I was starting a project by using Scrapy: I was essentially going to query everyday a page (a search page accessed by a POST call), getting the results (a list of links) and then get every single result page (download an XML file).

The project was paused but I'm thinking about restarting it, and I was thinking if something like diffbot or import.io could be useful for me.. any experience doing these kind of stuff?

Re: Upton: A Web Scraping Framework

#63

Post by gee_totes » Wed, Jul 24, 2013, 12:47 AM UTC

Shorel wrote: post #27

What I need is something that can scrape .NET sites with lots of weird and signed AJAX stuff just to populate a select control. I'm using node.js with several libraries and nothing has worked so far.

You might want to look at one of the headless web browsers, like phantom or casper,

http://phantomjs.org/ http://casperjs.org/

Re: Upton: A Web Scraping Framework

#64

Post by chuckd1356 » Wed, Jul 24, 2013, 11:15 AM UTC

ricardobeat wrote: post #29

Looks like a very nice integrated solution for data scientists and researchers. It's interesting to see the different shapes tools like this take, depending on their target users. I've been happy with node.js + cheerio[1] and it's simplistic jquery-like API: request 'http://website.com/list_of_stories.html', (err, body) -> $ = cheerio.load(body) callback $('#comments li a.commenter-name').map(cheerio::text) Plus if y…

I found using cheerio with request and phantom.js makes anything possible.

https://github.com/mikeal/request https://github.com/sgentle/phantomjs-node

Re: Upton: A Web Scraping Framework

#65

Post by Shorel » Thu, Jul 25, 2013, 8:37 PM UTC

gee_totes wrote: post #63

Shorel wrote: post #27

What I need is something that can scrape .NET sites with lots of weird and signed AJAX stuff just to populate a select control. I'm using node.js with several libraries and nothing has worked so far.

You might want to look at one of the headless web browsers, like phantom or casper, http://phantomjs.org/ http://casperjs.org/

Already tried that, and already failed.

Some JS in the page makes webkit die.

I think a headless Firefox is my only hope.