Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upton: A Web Scraping Framework

propublica.org

Post reply on HN
Print view View original

21–30 of 65 posts

Re: Upton: A Web Scraping Framework

#21

Post by jeremybmerrill »

nwienert wrote: post #19

Always used anemone with great results http://anemone.rubyforge.org/ Seems like it's not being actively developed, but again, never had a problem. Edit: realize this is focused on single page scraping with data extraction. Could use them nicely together in fact.

I had never seen this before. I will check it out, at least for inspiration.

I'd say "scraping" is a little more focused on extracting data from specific pages as opposed to ALL pages as in "spidering", but the two are certainly cousins if not siblings. Anemone would probably be good at the same sorts of tasks Upton is designed for (i.e. scraping data contained on multiple pages).

Re: Upton: A Web Scraping Framework

#22

Post by martin-adams »

darrhiggs wrote: post #20
martin-adams wrote: post #16

Earlier quoted context omitted.

It can be, I've just launched https://myshopdata.com where online retailers can scrape their own content for synchronising with third party marketplaces.

Your app looks really good. What tech did you use if you don't mind answering?

Thanks!

My Shop Data is all PHP & MySQL, with Slim, Twig & Bootstrap. The web scraping aspect is another product of mine (forgive the clunky homepage, I'm going to turn this into an API platform) - https://grabnotify.com

GrabNotify is Node.js, Mongo, PHP, Bootstrap and PhantomJS. The undocumented API allows you to create a web crawler but define a JavaScript algorithm to extract the data off the page. Some retailers have dropdowns which update stock, images, etc, so this crawler can simulate mouse events, etc. My Shop Data will supply a custom crawler algorithm for each e-commerce web site through the API.

And finally, I've written a HTML to Markdown translator to extract page descriptions but keep some formatting while being transferable to other systems that don't support HTML.

The whole legality issue of web scraping is an interesting one. I'm planning to position GrabNotify as a web crawler, page monitor and HTML -> data tool, but only if you own or have permission to scrape the original content but need a simple way to grab and monitor the HTML into data. I'm not really interested in building a business of scraping other people's content without their permission.

Re: Upton: A Web Scraping Framework

#25

Post by fourstar »

dome82 wrote: post #15

What about import.io? Anyone has experience using it? Link: http://import.io/

Is this somewhat familiar to embedly.com? That's what I currently use, and it seems to work fine, but am curious if that's the best thing to be using (I'm typically just grabbing the image thumbnail, but it'd be nice to grab some text if it exists too.

Re: Upton: A Web Scraping Framework

#27

Post by Shorel »

What I need is something that can scrape .NET sites with lots of weird and signed AJAX stuff just to populate a select control.

I'm using node.js with several libraries and nothing has worked so far.

Re: Upton: A Web Scraping Framework

#29

Post by ricardobeat »

Looks like a very nice integrated solution for data scientists and researchers. It's interesting to see the different shapes tools like this take, depending on their target users. I've been happy with node.js + cheerio[1] and it's simplistic jquery-like API:

    request 'http://website.com/list_of_stories.html', (err, body) ->
        $ = cheerio.load(body)
        callback $('#comments li a.commenter-name').map(cheerio::text)
Plus if you need to handle javascript/ajax, just replace that with jsdom/chimera with minor changes.

[1] http://npmjs.org/package/cheerio

Post reply on HN