Ask HN: Web crawler generator

#1

Post by villagefool » Wed, May 09, 2012, 12:21 PM UTC

I am looking for a web crawler generator, that I could prototype what I would like to crawl using a visual dom explorer type of interface, is there such a thing? anything recommended? Thanks

Re: Ask HN: Web crawler generator

#2

Post by skram » Wed, May 09, 2012, 1:13 PM UTC

Not sure if you're looking for a hosted solution (there are less of those) or a library to build upon (google "scraper"/"crawler" and the programming language of your choice).

Either way, check out ScraperWiki: http://scraperwiki.com

Hope this helps.

Re: Ask HN: Web crawler generator

#3

Post by villagefool » Wed, May 09, 2012, 2:09 PM UTC

skram wrote: post #2

Not sure if you're looking for a hosted solution (there are less of those) or a library to build upon (google "scraper"/"crawler" and the programming language of your choice). Either way, check out ScraperWiki: http://scraperwiki.com Hope this helps.

Interesting service, never heard of it. I am actually looking for a library/framework rather than service, I was hoping there is some "visual editor" for defining what to scrape/crawl and simplify the building process.

Re: Ask HN: Web crawler generator

#4

Post by skram » Wed, May 09, 2012, 3:01 PM UTC

villagefool wrote: post #3

skram wrote: post #2

Not sure if you're looking for a hosted solution (there are less of those) or a library to build upon (google "scraper"/"crawler" and the programming language of your choice). Either way, check out ScraperWiki: http://scraperwiki.com Hope this helps.

Interesting service, never heard of it. I am actually looking for a library/framework rather than service, I was hoping there is some "visual editor" for defining what to scrape/crawl and simplify the building process.

Well like I said, google or even a targetted search on Github/Sourceforge will reveal a TON of libs. As for a GUI... I don't recall any free ones coming up in my search a few months back.