Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

gregreda.com

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Web Scraping 101 with Python

#31

Post by chewxy »

kimagure wrote: post #20
samarudge wrote: post #6

PyQuery is pretty awesome ( https://pypi.python.org/pypi/pyquery ) Using Requests to download the document, pump it into PyQuery and you can use any jQuery style selectors to get text, attributes and all sorts of other stuff. Example; Here's how to scrape the hacker news homepage https://gist.github.com/samarudge/035ab8aaca224415cb49 (that code could probably be improved but I only spent a couple of minutes on it)

PyQuery seems to always be faster in my experience than BS4 (for ripping the same information). Anyone else have a similar experience?

Only on wellformed pages. There are many many many many many malformed pages on the internet. Even those that are created in 2013

Re: Web Scraping 101 with Python

#32

Post by creaktive »

thrownaway2424 wrote: post #5

But why python? Stop now and use perl. Perl's HTTP libraries are actually sane, unlike urllib[2], WWW::Mechanize is brilliant, and it's easier to throw in disgusting hacks in perl when you need them, which is constantly in the business of scraping the web.

I bet Perl-based scrapers run faster ;) http://news.ycombinator.com/item?id=5252581

Re: Web Scraping 101 with Python

#33

Post by ankimal »

sergiotapia wrote: post #14

Here are some awesome libraries I've used for HTML scraping: 1. Python - BeautifulSoup 2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website). 3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom e…

Also recommend Mechanize for Ruby (uses nokogiri under the covers).

http://mechanize.rubyforge.org/

http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites.

FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs the possibly altered DOM (after JS has run on the page) that you might see with Dev Tools/Firebug.

Re: Web Scraping 101 with Python

#34

Post by bjourne »

niggler wrote: post #3

For the pythonistas: what is the relationship between BeautifulSoup, lxml, urllib*, scrapy and mechanize?

Here are my highly opinionated opinions about their respective use cases:

* BeautifulSoup: It was the best scraping library ever until python-lxml came around and stole the show. Despite that the manual said BeautifulSoup gives you unicode, damnit! it had some long-standing bugs which it gave you strings or incorrectly decoded web pages. I wouldn't use it anymore because lxml is strictly superior.

* lxml: The king of scraping libraries. It is a big library so it can be hard to approach. It's actually a collection of markup parsers; lxml.html, lxml.etree and some more I've forgotten about. I almost exclusively use lxml.html since it "works" and can handle invalid markup without complaining. Use it like this: https://gist.github.com/mattoufoutu/823821 lxml can parse both using jQuery-style selectors with cssselect() and XPath 1.0 with the xpath() method. XPath is hard to learn, but once you get it you have a really powerful parsing tool which makes your life simpler.

* pyQuery: Easy to get started with and use. But not as powerful as lxml+XPath.

* urllib: Many of the modules in Python's standard library are there because they have been there for a long time. :) python-requests or httplib2 are the best libraries for http.

* scrapy: A framework for scheduling and supervising scraper spiders. It takes care of everything from downloading pages, following urls, concurrent requests, handling network errors to storing data in a database or generating csv files. It doesn't parse html itself, but delegates that task to lxml. Personally, I've found scrapy to be very good when your problem fits with how scrapy thinks scraping should be done. If you try to depart from the scrapy-way then scrapy suddenly feels very "frameworkish" and limiting. For example, i spent a lot of time trying to get it to support delta-scraping -- periodically scraping the same site, but only download new or changed data -- but it felt impossible getting scrapy to work the way I wanted.

* mechanize: Python port of the Perl module WWW::Mechanize. It's good for tasks like scripting logins. If you want to automate login to a site with a username and password, without having to care about session cookies, then mechanize is the ideal choice. Look elsewhere for an html parsing library.

* Scrapemark: It has a fun and clever approach to scraping. But once again, not as powerful as lxml.

Re: Web Scraping 101 with Python

#35

Post by dpearson »

sergiotapia wrote: post #14

Here are some awesome libraries I've used for HTML scraping: 1. Python - BeautifulSoup 2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website). 3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom e…

> Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;)

Having written code to both leverage a site's (private) API and to scrape that same site (when the private API stopped existing), I would much, much rather use an API. Yes, the scraper works, but the scraper's code is much messier, and JSON keys, for example, provide some documentation in their own right. Looking at the scraper months later, there's a lot more headscratching. Scraping will work, but it also will leave you searching for many little pieces of information that would be exposed by an API but aren't by a static site.

Still, I will agree that scraping is much easier; I've used jsdom (Node.js) extensively, and, for my use cases, it feels like working in a browser (full DOM, scripts, etc.).

Re: Web Scraping 101 with Python

#37

Post by ankimal »

nsp wrote: post #21

I'm not generally a huge fan of javascript, but phantomjs/casperjs are far and away the best tools I've used for scraping. Two features that stood out: 1. It's a headless WebKit browser, so it plays well with javascript and (sorta) flash. 2. It's easy to capture screenshots of the pages you're scraping, which is great for sanity checks later on. Http://phantomjs.org - the main engine Http://Casperjs.org - syntactic s…

They make scraping as easy as finding the right jquery selectors (once you inject jQuery onto the page) but can be very slow as compared to a vanilla HTML only scraper.

In my experience, a phantom/casper implementation could take upwards of 5-10 secs. to process a single page (almost 5-10x slower). This, even if you disable load of remote images and plugins.

Re: Web Scraping 101 with Python

#38

Post by pisarzp »

sbrother wrote: post #36

I'm surprised this article didn't mention scrapy. I had to do a lot of web scraping for a healthcare-related project last month and found scrapy incredibly fast and easy to use.

I have to admit that Scrapy is very fast, powerful and easy to use and scale. However, probably it's easier to start with BS, as Scrapy requires you to learn "Scrapy way of doing stuff". Furthermore, I find documentation to be a bit unpolished sometimes.

Still, Scrapy it's amazing and we use it a lot.

Re: Web Scraping 101 with Python

#39

Post by johnnyg »

ankimal wrote: post #33
sergiotapia wrote: post #14

Here are some awesome libraries I've used for HTML scraping: 1. Python - BeautifulSoup 2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website). 3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom e…

Also recommend Mechanize for Ruby (uses nokogiri under the covers). http://mechanize.rubyforge.org/ http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites. FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs th…

Mechanize is far and away the best and easiest way to scrape with Ruby until anything is rendered in javascript, which is explicitly not supported.

I tend to use Mechanized until I can't, then switch to Watir. Over time, I've found myself just strait up picking up Watir as it runs your browser directly and supports javascript rendering as a result.

Re: Web Scraping 101 with Python

#40

Post by darkarmani »

boyter wrote: post #9
niels wrote: post #7

This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.

This can actually sometimes be a feature. It makes it far less likely to have your IP banned. Its also a far more polite way to crawl someones site.

You could crawl a lot of different sites one page at a time. When I wrote a large distributed download system, I would use pycurl's bandwidth throttle and also store a 5 minute average of bandwidth per domain that would prevent other downloaders from saturating a domain.
Post reply on HN