Ask YC: What do you scrape? How do you scrape? • Hacker News Board

Re: Ask YC: What do you scrape? How do you scrape?

#31

Post by imrobotmaker » Wed, Apr 09, 2008, 9:17 PM UTC

I use curl, wget and links to retrieve data from sites and then I filter it with old sed and grep.

I created a mashup of AIM + Flicker.

If you use AIM 6 or AIM lite send a message to MyPictureBuddy

then send a message and enjoy.

basically You type a keyword and it gots to flicker and retrieves image information to display pictures right inside your AIM chat session.

I also have another Bot that parses HackerNews XML and then display it on the chat session. The bot name is

HackerNewsYC

Re: Ask YC: What do you scrape? How do you scrape?

#32

Post by ashu » Wed, Apr 09, 2008, 9:26 PM UTC

What: Banks. With: lib-www-perl.

Re: Ask YC: What do you scrape? How do you scrape?

#33

Post by m0nty » Wed, Apr 09, 2008, 9:42 PM UTC

alex_c wrote: post #28

m0nty wrote: post #16

Earlier quoted context omitted.

"Also AJAX certainly has changed the way a lot of screen scraping is done." I'd be interested in how you tackle this one. I've always used something like Perl/Curl/wget etc for scraping, but (like you say) JavaScript messes that up. I've had moderate success using GreaseMonkey and regexps in JavaScript code, but it's a bit fragile. I'm thinking of using GreaseMonkey + jQuery, since that should allow me to select DOM…

Even though it's actually a testing tool, you might have some luck with Canoo Webtest + Groovy ( http://webtest.canoo.com ). Webtest uses HtmlUnit which has pretty good Javascript support, and means you don't have to mess with regexps to get around the document structure, and Groovy lets you use an actual programming language rather than the awkward Ant-based syntax of Webtest. It takes some getting used to, and I ha…

Thanks, I'll give it a try. I'm collaborating on a project which involves getting info from online financial markets, btw, but it's getting held up because of this scraping problem. So new ideas might help get it moving again.

Re: Ask YC: What do you scrape? How do you scrape?

#34

Post by herdrick » Wed, Apr 09, 2008, 9:56 PM UTC

HtmlPrag turns any HTML into nice s-expressions. It's a Scheme library. http://www.neilvandyke.org/htmlprag/

I've used it a lot - it's really great.

Re: Ask YC: What do you scrape? How do you scrape?

#35

Post by inovica » Wed, Apr 09, 2008, 10:46 PM UTC

friism wrote: post #27

Scraping EU public procurement contracts from the "Tenders Europa Daily" database ( http://ted.europa.eu/ ). There's more than a million documents with each document requiring up to two requests. Been at it for several weeks with a multithreaded scraper and we're almost through. Using Solvent (simile.mit.edu/solvent/) to generate xpath expressions and HtmlAgilityPack (www.codeplex.com/htmlagilitypack) to run the xpat…

Be careful here. The content is actually copyrighted. Whilst you can scrape it their T&Cs expressly forbid it. They sell licenses to access this information - the license is NOT expensive and they provide direct access to all the data in XML.

Re: Ask YC: What do you scrape? How do you scrape?

#36

Post by inovica » Wed, Apr 09, 2008, 10:56 PM UTC

We are working on a lot of scraping and analysis and here are a few links that you might be interested in if you are using Python:

http://pyro.sourceforge.net/

http://pyprocessing.berlios.de/

http://www.sqlalchemy.org/

http://codespeak.net/lxml/

http://nltk.org/index.php/Main_Page

The biggest hurdle is in understanding how to navigate through a complex site - such as a forum, real estate etc. We have created a visual tool for this however there are other methods. Look at dapper.net as this is useful.

I am wondering if there could be some collaborative effort from the minds on this site to create something unique and groundbreaking

Re: Ask YC: What do you scrape? How do you scrape?

#37

Post by johnb » Wed, Apr 09, 2008, 11:33 PM UTC

I'm a big fan of using Hpricot + Ruby. I'd say the sites I had been scraping but I doubt my old client wants it to come out :|

To get the most bang for my buck (developer time wise) I would visit each site with firebug in inspect mode, hover the data I want to extract. From there I figure out how I would style that element, and because Hpricot supports CSS selectors I've straight away got a method for pulling that data out of the page.

Re: Ask YC: What do you scrape? How do you scrape?

#38

Post by yters » Wed, Apr 09, 2008, 11:48 PM UTC

anyone have success with emacs and w3? I haven't given it a shot yet, but seems like its interactive nature might be useful.

Re: Ask YC: What do you scrape? How do you scrape?

#39

Post by nreece » Thu, Apr 10, 2008, 3:06 AM UTC

Our startup, Feedity - http://feedity.com , generates/creates RSS web feeds from virtually any webpage, for the purpose of content tracking and mashup data reuse.

We scrape public webpages (with an option for content owners to restrict access), and we use the .NET Framework' in-built socket implementation (System.Net namespace) for fetching remote content.

Our biggest frustration was to deal with invalid charset/content encoding of the source webpages. But we resolved it using a custom module. Now everything we parse is unicode (utf-8)!

The collest hack we've encountered while scraping is utilizing the Conditional GET behavior using the HTTP If- Modified-Since header.

Re: Ask YC: What do you scrape? How do you scrape?

#40

Post by sheriff » Thu, Apr 10, 2008, 4:38 AM UTC

My startup, http://www.FuseCal.com (previously discussed at http://news.ycombinator.com/item?id=146134), scrapes calendar events out of web pages and into your personal calendar. In the general case, we don't know anything about the layout of the page before trying to extract the events, so there's something of a classification problem first.