Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask YC: What do you scrape? How do you scrape?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Ask YC: What do you scrape? How do you scrape?

#23

Post by fallentimes »

thorax wrote: post #6

I use BeautifulSoup when needed for simple scraping. My biggest frustrations, right now, are really around getting data from lots of different websites in subtly varied forms. This is a tough problem to automate. I certainly haven't found any tools that make it simple. I'd be happy with a 50% correctness rate, looking for very loose patterns. I just haven't found a tool and, while I have some ideas for how to do it,…

Getting just the recipe would be the hardest part, but it's still doable. Once you figure out that you're currently parsing a recipe (via keywords, close matching, whatever) you could fan out and look for common start/end tags like

, , etc. If you use something like Beautiful Soup you could do this pre-parse instead of post-parse and eliminate a lot of extra stuff (no recipes in the tag, etc.)

After that it just becomes an issue of removing the cruft around the recipe. I would start with common stuff: splitting things up by
or inner

since if someone is gonna have something before / after their recipe (say, on a forum) it'll be split up with blank lines somehow (well, usually). This will be another time to use things like close matching and teaching the algorithm what it gets right/wrong so it can weigh things as recipe/not better in the future.

If you do all this and add more specific edge cases as time goes on, I think you'd be able to maintain a 50% correctness rate pretty easily.

Edit: And it'd be much cheaper than a neural network ;)

Re: Ask YC: What do you scrape? How do you scrape?

#24

Post by jharrison »

I used Mechanize and Hpricot on a project recently to create a sort of poor-man's API. My client is a performing arts organization that wanted a new website but they already had a (dreadful) internally-hosted site for selling tickets.

In order to keep website users from having 2 accounts I created an interface that scrapes the sign in, sign up, lost password, change password, and couple other screens of the internal system. So when users come to the website and "login" they're actually logging in to the internal system and I just record their session from the internal system so I can masquerade as them as they go about their business.

It's not going to support 100s of connections per second but it gets the job done for their traffic levels (36,000 views the first day of launch).

Re: Ask YC: What do you scrape? How do you scrape?

#25

Post by mrtron »

dangoldin wrote: post #14

I come from a Perl background so I've been using HTML::TreeBuilder and XML::TreeBuilder to do my parsing. It will basically load an HTML/XML file into it's own tree structure and give you an easy way to go through it. By knowing how each site names their divs/classes I am able to scrape. I took a quick glimpse at beautiful soup and it seems to be doing something similar - someone let me know if this is correct.

Yes. You can even regex search through the tree. Weeeeee!

BeautifulSoup is nothing unique, but it can handle malformed data that saves you a ton of hassle.

Re: Ask YC: What do you scrape? How do you scrape?

#26

Post by hooande »

I've done my share of screen scraping, gathering all different kinds of data. Movies, sports, finance, you name it. Here are three things I can tell you:

1. Take the time to get very familiar with regular expressions. If you think you know your regex pretty well, go to the docs or get a book and find three things you don't understand and understand them fully. Then find three more.

2. The data doesn't have to be perfect. In most cases you can clean it after you've stored it. It's generally better to get more than you think you might need (in terms of data or html/formatting around the data) and then go back and clean it later

3. Generally, my most successful data mining algorithms involve a lot of hacks. There are very few clean formulas...usually I have to play with the data for awhile and fix a lot of one offs and special cases and then it ends up coming out ok

Re: Ask YC: What do you scrape? How do you scrape?

#27

Post by friism »

Scraping EU public procurement contracts from the "Tenders Europa Daily" database (http://ted.europa.eu/). There's more than a million documents with each document requiring up to two requests. Been at it for several weeks with a multithreaded scraper and we're almost through. Using Solvent (simile.mit.edu/solvent/) to generate xpath expressions and HtmlAgilityPack (www.codeplex.com/htmlagilitypack) to run the xpath on the downloaded html with regexps as the topping. They're a match made in heaven (http://www.itu.dk/~friism/blog/?p=40).

The login procedure is gothic and took a lot of wiresharking to figure out. .Net has pretty good scraping-support in the WebClient and HttpWebRequest classes found in the System.Net namespace.

Will publish results soon... :-)

Re: Ask YC: What do you scrape? How do you scrape?

#28

Post by alex_c »

m0nty wrote: post #16
foodawg wrote: post #12

I do all my screen scraping with PHP, curl, and some regex. Previously I used plain PHP. I use it to scrape television listing data ( http://ktyp.com/rss/tv/ was my old site, and http://code.google.com/p/listocracy/ ) and more recently to scrape resume data from job posting websites for a (YC-rejected :P ) side project I'm working on. The hardest part I've encountered with scraping is odd login and form setups. For e…

"Also AJAX certainly has changed the way a lot of screen scraping is done." I'd be interested in how you tackle this one. I've always used something like Perl/Curl/wget etc for scraping, but (like you say) JavaScript messes that up. I've had moderate success using GreaseMonkey and regexps in JavaScript code, but it's a bit fragile. I'm thinking of using GreaseMonkey + jQuery, since that should allow me to select DOM…

Even though it's actually a testing tool, you might have some luck with Canoo Webtest + Groovy (http://webtest.canoo.com). Webtest uses HtmlUnit which has pretty good Javascript support, and means you don't have to mess with regexps to get around the document structure, and Groovy lets you use an actual programming language rather than the awkward Ant-based syntax of Webtest. It takes some getting used to, and I haven't used it for web scraping, but it's a pretty powerful combination.
Post reply on HN