Live data from Hacker News

Web Scraping 101 with Python

gregreda.com

51–60 of 80 posts

Re: Web Scraping 101 with Python

#51
post #42
post #17

Why is he using lxml as the parser and not just the one built into BS?

BeautifulSoup uses regular expressions! http://stackoverflow.com/questions/1732348/regex-match-open-... Like many here point out, lxml is a fast and versatile library that could be used for this alone without BS. lxml.html can parse HTML and lxml also has support for using HTML5 parser from html5lib that deals with broken HTML in the standardized way.

> BeautifulSoup uses regular expressions!

Holy hell, you're right.

http://bazaar.launchpad.net/~leonardr/beautifulsoup/bs4/view...

Re: Web Scraping 101 with Python

#53
post #39
post #33

Earlier quoted context omitted.

Also recommend Mechanize for Ruby (uses nokogiri under the covers). http://mechanize.rubyforge.org/ http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites. FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs th…

Mechanize is far and away the best and easiest way to scrape with Ruby until anything is rendered in javascript, which is explicitly not supported. I tend to use Mechanized until I can't, then switch to Watir. Over time, I've found myself just strait up picking up Watir as it runs your browser directly and supports javascript rendering as a result.

How is performance with Watir? With casperjs a page takes me on an avg. 5-10 secs. to process.

Re: Web Scraping 101 with Python

#54
post #48
post #33

Earlier quoted context omitted.

Also recommend Mechanize for Ruby (uses nokogiri under the covers). http://mechanize.rubyforge.org/ http://phantomjs.org/ or "its easier to deal with cousin" http://casperjs.org/ for very client heavy sites. FWIW, you can get away with HTML only scrapers most of the time, you just need to look harder to find all the data. Totally recommend using "View page source" as that would always give you the original HTML vs th…

I recommend Selenium before I'd recommend PhantomJS in situations where Mechanize/Nokogiri don't cut the mustard, I've found Selenium scripts much easier to comprehend, modify, and maintain over time than the PhantomJS scripts.

Check out casperjs, it should make life easier. Phantomjs by itself is extremely cumbersome in my experience.

Re: Web Scraping 101 with Python

#55
For those who have been trying to scrape pages that make AJAX requests, something you can not do with BeautifulSoup alone - I would recommend using Selenium Server (http://docs.seleniumhq.org/). This allows you to automate a real browser (so your scraping requests look like real page requests, not robots).

In addition, tools exist for Selenium that let you scale up easily. You can use Selenium Grid 2 (https://code.google.com/p/selenium/wiki/Grid2) to run multiple browser instances in parallel. This is very beneficial for web scraping or automated UI testing.

Re: Web Scraping 101 with Python

#56
post #26
post #19

BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSo…

In my experience lxml.html is much better.

I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent.

It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV

Re: Web Scraping 101 with Python

#57
post #34
post #3

For the pythonistas: what is the relationship between BeautifulSoup, lxml, urllib*, scrapy and mechanize?

Here are my highly opinionated opinions about their respective use cases: * BeautifulSoup: It was the best scraping library ever until python-lxml came around and stole the show. Despite that the manual said BeautifulSoup gives you unicode, damnit! it had some long-standing bugs which it gave you strings or incorrectly decoded web pages. I wouldn't use it anymore because lxml is strictly superior. * lxml: The king of…

Thank you. This is why I come back to HN: for pretty much every technical question I have, there's someone with experience with many alternatives and can give their opinions :)

Re: Web Scraping 101 with Python

#58
Not to rain on the parade of this post (I'm in support of more people learning to scrape, and more services out there giving us easier to access data). I'm someone who loves web scraping, but I'm also someone who believes that if you don't know what the library is doing, you shouldn't be using it.

You can give a brief overview of how to use it and what to look for in the page to extract from, but you're giving a very simple cheat sheet to people that may not understand HTML (trust me they exist... unfortunately). As soon as your example breaks, or they reach a limitation with the library, they are going to throw their arms in the air and deem the library broken, or the task impossible to do because the example said it would work. The only reason I'm writing this is that I know of these sorts of people, I deal with them on a regular basis, and I have to explain to them every time to look at what they are doing on a lower level to get a better understanding of their problem to find the solution.

These sorts of people will stumble across this article after their bosses told them "We need to pull Company X's product information into our sales screens so that we can compare the competitions prices while making our price adjustments". Knowing that they don't even have a clue on how to do that, they will Google for it and retrieve this article. With no experience, and an boss behind them, they will just blindly use it and pray that it works, but due to their inexperience with the subject at hand they will fail.

Sorry to be so negative, I just had to say that. It's the same as any other tutorial out there, just Scraping is something that I feel you need to know what you're doing before you do it.

Personally, I write my own scrapers from scratch (or using libraries I have written over time to make certain aspects less painful) for years. I know, I know, there is a myriad of ready-to-go libraries out there that will do the same thing and probably better for me, but where's the challenge. Sure if you're time restricted, then go forth and grab a library and start scraping, but please at least try to understand what you are doing at a lower level.

Re: Web Scraping 101 with Python

#59
post #38

I'm surprised this article didn't mention scrapy. I had to do a lot of web scraping for a healthcare-related project last month and found scrapy incredibly fast and easy to use.

I have to admit that Scrapy is very fast, powerful and easy to use and scale. However, probably it's easier to start with BS, as Scrapy requires you to learn "Scrapy way of doing stuff". Furthermore, I find documentation to be a bit unpolished sometimes. Still, Scrapy it's amazing and we use it a lot.

Scrapy is awesome and we have been using it without any problem so far.

Re: Web Scraping 101 with Python

#60
post #58

Not to rain on the parade of this post (I'm in support of more people learning to scrape, and more services out there giving us easier to access data). I'm someone who loves web scraping, but I'm also someone who believes that if you don't know what the library is doing, you shouldn't be using it. You can give a brief overview of how to use it and what to look for in the page to extract from, but you're giving a very…

I'm definitely not advocating for people not understanding the problem they want solved.

That said, your post sounds empty. Can you elaborate on why your own scrapers that you write from scratch make it all better? How do you your scrapers deal with encode detection, broken html, content prioritization and so forth?

I don't like the current options we've got in pythonland, but just writing: "this sucks, so I write my own" sounds like an ego trip. Can you describe in detail what BeautifulSoup (or lxml which is usually a better option) is doing wrong at the lower level and how your scripts are making it better?

Post reply on HN