Ask YC: Any crawler experts out there?

Hi. Wondering if there are any crawler experts on here who can help me. We're wanting to create a crawler to visit some sites that have forms, lists of items and detail of items. They're all in the real estate market and we want to capture the properties and pull out the latest ones. I'm being told that we need to create a specific crawler for each site, but I was wondering if we could create a generic crawler that has some kind of plug-in or pattern matching file (that we build manually) for each site. Anyone who is super-skilled in this area - I'd appreciate some advice. We're using Python also. One caveat - I'm not the tech guy as I tried to program and failed, but I do understand what we need and have a very good understanding of technology, I'm just inept at taking my ideas and doing anything with them :)

You can build a generic crawler that pulls pages from sites quickly and then process the pages offline with whatever lnguage you'd like. It's better to have a distributed way of doing things. Plus, there are standards that you need to comply with when crawling someone's website., like not crawl them too fast, or to check their robots.txt file to make sure that you're crawling "allowable" pages. Then once you've pulled their data off, you process it offline and do whatever you need to do with the data. It's not a simple procedure, but it's do-able if you want to spend some time doing it properly.

MLS. :)

I work for http://streeteasy.com . We are experts in this area - especially in real estate ;) Feel free to contact me at ch AT streeteasy.com. It is not an easy task to build what you are looking for. We've been building and improving our system for two years now (ruby on rails.) Scraping the data is just one part of the problem. Validating the data is also a big issue. These sites often have incorrect or stale information. MLS's are good, but they may have restrictions on what you can do with the data, or (as in NYC) they may not even exist.

I've developed a Twisted Python crawler that does something very similar to that. The possibility that it would work well seemed dubious at first, but I've been pleasantly surprised with the results.

Email me at johnwehr@gmail.com - I'd be happy to discuss the technology and progress I've made so far.