Please contact latone at gmail.com and we can discuss.
Ask YC: Any crawler experts out there?
11–17 of 17 posts
Re: Ask YC: Any crawler experts out there?
#12use beautiful soup (best python scraping system I know of). Maybe combine it with mechanize for navigating between the pages.
Don't try to create your own pattern-matching file. Write a generic crawler class, subclass it for each site. In the end you should just need to write a couple of short site-specific functions for each site.
It doesn't take long once you get going - that is, until you run into sites that are unnavigable piles of javascript and unstructured html.
Re: Ask YC: Any crawler experts out there?
#13use beautiful soup (best python scraping system I know of). Maybe combine it with mechanize for navigating between the pages. Don't try to create your own pattern-matching file. Write a generic crawler class, subclass it for each site. In the end you should just need to write a couple of short site-specific functions for each site. It doesn't take long once you get going - that is, until you run into sites that are u…
Thank you. I'll look into these. I'm constantly amazed by the power of python - especially in comparison to PHP which is my background
Re: Ask YC: Any crawler experts out there?
#14If your environment supports Ruby, I have lots of experience with a great library called Scrubyt... yickster at gmail
Re: Ask YC: Any crawler experts out there?
#15I'm with screen-scraper, (http://www.screen-scraper.com/), and we've dealt a lot with scraping real estate data. Building a generic crawler for this kind of thing is quite a bit more complicated than it might seem. You might give our software and services a look, though. Our app integrates quite nicely with Python.
Re: Ask YC: Any crawler experts out there?
#16You might talk to the guys at New Idea Engineering about their xpump technology http://www.ideaeng.com/ds/xpump.html
I used it a few years ago to process all of the hardware data sheets on the Cisco website and extract 13 parameters such as height, width, depth, weight, power consumption (AC and DC), etc...Because Cisco's products come from many different acquisitions the datasheets were in many different formats, which sounds similar to your problem.
One point I would make is that a fast crawler is not always the best for this type of application: crawling at about the speed a user would click on pages is more friendly to a site and less likely to have them take steps to block your access.
Re: Ask YC: Any crawler experts out there?
#17I've been working on this for a little while now. You can definitely write a plug-in or pattern matching file for each site. Building a specific crawler for each website doesn't make sense.
The custom bits you need are then ones that fill the form and then extract the results. For scraping results ruby's Scrubyt is best as you can write templates for each type of page.