Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upton: A Web Scraping Framework

propublica.org

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Upton: A Web Scraping Framework

#41

Post by look_lookatme »

dangoldin wrote: post #11
terhechte wrote: post #2

For the past couple of years, I've always done any web scraping with trusty python + beautiful soup or elementtree. I've recently started doing it with Clojure + Enlive (mostly as an excuse to use clojure for less academic excercises) but I really like it. From that perspective, Upton looks pretty cool, especially the debug mode.

Have you taken a look at Scrapy ( http://scrapy.org/ )? My evolution has been from Perl to Python and I recently did a project with Scrapy that left me pretty happy.

Scrapy is really awesome. It's a soup to nuts queuing, fetching and extraction workflow tool and if I had to start a larger than trivial project (like an rss reader or shopping aggregation site) I would base my spider toolchain on it.

Re: Upton: A Web Scraping Framework

#42

Post by joshfraser »

Recently I've been using Fake Browser (fakeapp.com) for web scraping. While it's inefficient for large jobs, it's awesome for hacking together quick scripts. With Fake you can write your scraper in JavaScript and it run in an actual browser so it's very visual process. It's great for instances where you want to get past complicated authentication systems without writing code. Just sign in manually and start your script.

Re: Upton: A Web Scraping Framework

#43

Post by hobonumber1 »

jeremybmerrill wrote: post #7
hobonumber1 wrote: post #3

I use YQL to do web scraping. It lets me do something like this: `select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"` Could you elaborate on the benefits of using Upton instead of this?

AFAICT, YQL can only handle scraping individual pages that way. Upton can scrape a whole set of pages. If you have a page that lists the pages you're interested in; suppose you're interested in HN commenters on front page posts, you could specify the front page URL and a selector for links to comment pages, and Upton would automatically scrape those pages and return them to you. Upton could even write the commenter n…

Makes sense! Thanks for clarifying that.

Re: Upton: A Web Scraping Framework

#44

Post by kanzure »

dangayle wrote: post #36
kanzure wrote: post #12

Earlier quoted context omitted.

You don't even need phantomjs, really. You can use python + webkitgtk+ through the gobject bindings. The problem with phantomjs is that it's using an old version of webkit from an old version of qtwebkit from qt 4.8, whenever that was released. By comparison, webkitgtk+ can be compiled from upstream webkit whenever you please. If you insist on controlling webkit through javascript, you can use gnome-seedjs. But this…

Do you have any other resources about using webkitgtk with python for this sort of purpose?

No, I haven't found a definitive tutorial or reference (even for webkit). In general, look around for "import gi.repository.webkit" and you will find relevant things. I am not very sure how the other phantomjs developers are learning webkit things.. probably just reading code.

Re: Upton: A Web Scraping Framework

#45

Post by miket »

Instead of (or in addition to) manually creating scrapers, you can use Diffbot to automatically extract this type of information from news articles using computer vision: http://diffbot.com/products/automatic/article. It also allows you to create rules with a WYSIWYG editor: http://diffbot.com/products/custom/

Re: Upton: A Web Scraping Framework

#46

Post by jeremybmerrill »

danso wrote: post #30

Cool library, Jeremy...another Ruby scraping framework you might want to check out and improve upon is Artsy's Spidey: https://github.com/joeyAghion/spidey Has a similar approach but also leaves storage (and caching) up to the end-user.

Thanks, Dan. I'll check that out, especially for good ideas on how to solve things I haven't solved yet. Spidering and scraping seem to be very related, but not quite the same -- and I admittedly know nothing about spidering.

Re: Upton: A Web Scraping Framework

#47

Post by dangoldin »

look_lookatme wrote: post #41
dangoldin wrote: post #11

Earlier quoted context omitted.

Have you taken a look at Scrapy ( http://scrapy.org/ )? My evolution has been from Perl to Python and I recently did a project with Scrapy that left me pretty happy.

Scrapy is really awesome. It's a soup to nuts queuing, fetching and extraction workflow tool and if I had to start a larger than trivial project (like an rss reader or shopping aggregation site) I would base my spider toolchain on it.

Yea - I was definitely impressed by it and I just got started. It felt as if I was able to get rid of all the boilerplate and just focus on getting the next page that needed to be crawled and the information to extract.

Re: Upton: A Web Scraping Framework

#49

Post by paulhauggis »

kingkool68 wrote: post #40

PHP SimpleDOM Class makes scraping just like working with jQuery on the backend. http://simplehtmldom.sourceforge.net/

I used this for many years, but the memory footprint is terrible.

I would recommend querypath. Very small footprint and takes a fraction of the cpu time.

Post reply on HN