For those who want to use a java based solution, I invite you to check out my open source block tolerant (IP Blocking) web scraper that runs on top of aws and rackspace, called Tales. Tales is designed to be easy to deploy, configure, and manage. With Tales you can scrape 10s or even 100s of domains concurrently.
Web Scraping 101 with Python
11–20 of 80 posts
Re: Web Scraping 101 with Python
#12PyQuery is pretty awesome ( https://pypi.python.org/pypi/pyquery ) Using Requests to download the document, pump it into PyQuery and you can use any jQuery style selectors to get text, attributes and all sorts of other stuff. Example; Here's how to scrape the hacker news homepage https://gist.github.com/samarudge/035ab8aaca224415cb49 (that code could probably be improved but I only spent a couple of minutes on it)
I definitely recommend this for people used to the jquery syntax. Requests + PyQuery took no time at all to learn and did everything I needed for some basic page crawling.
Re: Web Scraping 101 with Python
#13I've used Beautiful Soup (BS) and Scrapy BS is a _lot_ slower than Scrapy, although you can probably get up and running with BS first (it also more noob friendly imho since you don't have to learn yet another framework).
Learning Scrapy was made easier after some experience with BS.
Re: Web Scraping 101 with Python
#14Here are some awesome libraries I've used for HTML scraping:
1. Python - BeautifulSoup
2. Ruby - Nokogiri (use in conjunction with Watir if you're scraping a very client-heavy website).
3. C# - HtmlAgilityPack in conjunction with ScrapySharp (there's a nuget package for both) - I highly recommend ScrapySharp because it allows you to query elements using a very familiar Css selector type similar to how you query dom elements in jQuery. :)
Scraping online content is so simple these days, if a website doesn't offer an API you still have alternatives ;)
Re: Web Scraping 101 with Python
#15This will only be a good approach if you are going to scrape a small amount of pages. The problem is using synchronous requests, as this blocks the crawler until a request has finished. Using asynchronous requests such as supported by twisted (and scrapy) will allow you to crawl a lot faster using the same resources.
I've heard this from countless people who have read the post. It's definitely made me want to look into Scrapy.
Re: Web Scraping 101 with Python
#16If you are interested in learning to build scrapers, I highly recommend this book: http://www.webbotsspidersscreenscrapers.com/ , the code is all PHP so it is very approachable.
Re: Web Scraping 101 with Python
#17Why is he using lxml as the parser and not just the one built into BS?
Re: Web Scraping 101 with Python
#18Why is he using lxml as the parser and not just the one built into BS?
lxml is superior to BS. Most of the elementtree API is implemented by lxml too so it's compatible with BS - not sure why he's using BS when everything is built into lxml though and things like PyQuery and/or XPath parsing are available.
Re: Web Scraping 101 with Python
#19BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSoup is fine for small jobs but if you were making a web-crawler for example look elsewhere it is totally inadequate.
Re: Web Scraping 101 with Python
#20PyQuery is pretty awesome ( https://pypi.python.org/pypi/pyquery ) Using Requests to download the document, pump it into PyQuery and you can use any jQuery style selectors to get text, attributes and all sorts of other stuff. Example; Here's how to scrape the hacker news homepage https://gist.github.com/samarudge/035ab8aaca224415cb49 (that code could probably be improved but I only spent a couple of minutes on it)
PyQuery seems to always be faster in my experience than BS4 (for ripping the same information). Anyone else have a similar experience?