Live data from Hacker News

Simple web-crawler for Node.js

Post reply on HN

Print view View original

1–2 of 2 posts

Simple web-crawler for Node.js

#1

Post by DamonOehlman » Tue, May 15, 2012, 2:19 AM UTC

Simple web-crawler for Node.js
github.com

Re: Simple web-crawler for Node.js

#2

Post by zoowar » Tue, May 15, 2012, 2:24 AM UTC

Show some respect for the web and add robots.txt and sitemap.xml support.

Post reply on HN