Ask YC: How to web scrape 100s of sites/forms?

hi...

a potential research app requires the scraping of a few hundred websites/forms and diving into the child links to obtain the linked/parent structure ie. company->dept->title->name.

in this case, this would involve going 4 levels deep, and getting the required information.

so, does anyone know of a method/app/company that can be used to accomplish this. orm am i going to have to figure out how to get a number of cheap guys to write a bunch of python scripts!!

thanks

Or get one expensive guy to write you a script that

writes the scripts to scrape the sites by scraping the sites to read the structure to write the scripts to scrape the sites.

Libraries like mechanize and hpricot are shrinking the curve for scraping tasks. That's not to say it is easy, but it should not take a bunch of people working on it. One good developer with proper experience would be ample in my opinion.