Upton: A Web Scraping Framework
propublica.org
Upton: A Web Scraping Framework
1–10 of 65 posts
Re: Upton: A Web Scraping Framework
#2For the past couple of years, I've always done any web scraping with trusty python + beautiful soup or elementtree. I've recently started doing it with Clojure + Enlive (mostly as an excuse to use clojure for less academic excercises) but I really like it.
From that perspective, Upton looks pretty cool, especially the debug mode.
Re: Upton: A Web Scraping Framework
#3I use YQL to do web scraping. It lets me do something like this:
`select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"`
Could you elaborate on the benefits of using Upton instead of this?
Re: Upton: A Web Scraping Framework
#4For the past couple of years, I've always done any web scraping with trusty python + beautiful soup or elementtree. I've recently started doing it with Clojure + Enlive (mostly as an excuse to use clojure for less academic excercises) but I really like it. From that perspective, Upton looks pretty cool, especially the debug mode.
[deleted]
Re: Upton: A Web Scraping Framework
#5I use YQL to do web scraping. It lets me do something like this: `select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"` Could you elaborate on the benefits of using Upton instead of this?
I've always used this method:
http://railscasts.com/episodes/190-screen-scraping-with-noko...
As well as Mechanize when working with sites that require session cookies and all that.
I'm wondering too what the advantages of Upton are?
Re: Upton: A Web Scraping Framework
#6Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.
Re: Upton: A Web Scraping Framework
#7I use YQL to do web scraping. It lets me do something like this: `select * from data.html.cssselect where url="www.yahoo.com" and css="#news a"` Could you elaborate on the benefits of using Upton instead of this?
AFAICT, YQL can only handle scraping individual pages that way.
Upton can scrape a whole set of pages. If you have a page that lists the pages you're interested in; suppose you're interested in HN commenters on front page posts, you could specify the front page URL and a selector for links to comment pages, and Upton would automatically scrape those pages and return them to you.
Upton could even write the commenter names to a CSV for you with just a filename and a CSS selector/XPath expression.
It's not stuff you couldn't do with YQL or Python/BeautifulSoup. But it's stuff that I didn't want to have to write over and over each time I wrote a new scraper.
Re: Upton: A Web Scraping Framework
#8Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.
Nope. Sorry. :)
While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
Re: Upton: A Web Scraping Framework
#9For the past couple of years, I've always done any web scraping with trusty python + beautiful soup or elementtree. I've recently started doing it with Clojure + Enlive (mostly as an excuse to use clojure for less academic excercises) but I really like it. From that perspective, Upton looks pretty cool, especially the debug mode.
Thanks, I hope you like it.
Re: Upton: A Web Scraping Framework
#10Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.
There's phantomjs for that. Though I've actually had the most success with perl's WWW::Mechanize::Firefox and a headless X server.