Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upton: A Web Scraping Framework

propublica.org

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Upton: A Web Scraping Framework

#53

Post by inovica »

This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this is a crawling thread I hope you don't mind me asking some advice on where to take it from here :)

Here's how it currently works:

1) It has a queue of domains that I have pre-processed. For the initial purposes I've restricted it to pages that I think are ecommerce based on $ signs, add to cart/basket type links etc

2) There is a visual tool that I then use to select certain parts of the page - eg price, product, image etc. I save these out as xpaths

3) Once I have done one URL I send a crawler to that domain and extract other pages that fit the profile of an ecommerce page and try to use the same mapping as number 2 above to extract the data

I have done a small video to show it in action:

http://www.screencast.com/t/riB3iiVMiSk

I'm not sure if I'm doing this the right way. If a site/page changes structure then I may have to re-map the data. I was hoping that someone would have some pointers for me in terms of any other ways to do this. Also with Javascript-heavy sites I've had some problems

If anyone has any knowledge of screen scraping, where it can be done more automatically, I'd really appreciate a steer!

Re: Upton: A Web Scraping Framework

#54

Post by regularfry »

inovica wrote: post #53

This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this is a crawling thread I hope you don't mind me asking some advice on where to take it from here :) Here's how it currently works: 1) It has a queue of domains that I have pre-processed. For the initial purposes I've restricted it to pages that I think are ecomm…

I've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful.

One approach I found absolutely vital was to have a rewriting, caching proxy between the crawler and the upstream site. This proxy allowed me to rewrite the page content into something much simpler for the crawler to get to grips with (RSS or Atom, say). I used Celerity (http://celerity.rubyforge.org/) with a hacked-on Mechanize API to do the rewriting, which let me handle JS-heavy pages almost as easily as static HTML ones. My original inspiration for this was _why's Mousehole (the source for which is here: https://github.com/evaryont/mousehole, I've got no idea if it runs on recent Rubies).

The proxy also gives you somewhere to raise an alert if, all of a sudden, your scraping fails because of an upstream change.

One tool I always intended to make some use of, but never got round to, was Ariel: http://ariel.rubyforge.org/. It looks like it ought to be able to totally remove the need to manually extract xpaths.

Re: Upton: A Web Scraping Framework

#55

Post by draegtun »

valtron wrote: post #51

Web::Scraper is a really good one for perl. ( http://search.cpan.org/~miyagawa/Web-Scraper-0.37/lib/Web/Sc... ).

Typically with Perl there is more than one module for this :)

- pQuery | https://metacpan.org/module/pQuery

- Mojo::UserAgent | https://metacpan.org/module/Mojo%3a%3aUserAgent

- Scrappy | https://metacpan.org/release/Scrappy

- Web::Query | https://metacpan.org/module/Web%3a%3aQuery

- Web::Magic | https://metacpan.org/module/Web%3a%3aMagic

Above are specifically for scraping but one shouldn't forget WWW::Mechanize & LWP.

My preference over last few years is with pQuery. However Web::Query is Tokuhiro's pQuery improvement and Mojo::UserAgent looks very nifty.

Re: Upton: A Web Scraping Framework

#56

Post by inovica »

regularfry wrote: post #54
inovica wrote: post #53

This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this is a crawling thread I hope you don't mind me asking some advice on where to take it from here :) Here's how it currently works: 1) It has a queue of domains that I have pre-processed. For the initial purposes I've restricted it to pages that I think are ecomm…

I've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful. One approach I found absolutely vital was to have a rewriting, caching proxy between the crawler and the upstream site. This proxy allowed me to rewrite the page content into something much simpler for the crawler to get to grips with (RSS or Atom, say). I used Celerity ( h…

Thanks for this. I'll check these out

Re: Upton: A Web Scraping Framework

#57

Post by Walkman »

inovica wrote: post #53

This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this is a crawling thread I hope you don't mind me asking some advice on where to take it from here :) Here's how it currently works: 1) It has a queue of domains that I have pre-processed. For the initial purposes I've restricted it to pages that I think are ecomm…

Check out Scrapy http://scrapy.org/

Re: Upton: A Web Scraping Framework

#59

Post by wslh »

a8da6b0c91d wrote: post #10
jliechti1 wrote: post #6

Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.

There's phantomjs for that. Though I've actually had the most success with perl's WWW::Mechanize::Firefox and a headless X server.

Or you can use a full browser with extension to do the scraper. In this way you are up to date with the latest browser release.
Post reply on HN