Using Node.js and JQuery to Crawl Public Tweets • Hacker News Board

Re: Using Node.js and JQuery to Crawl Public Tweets

#11

Post by thegoleffect » Mon, Aug 06, 2012, 2:12 AM UTC

danso wrote: post #10

Does Node have anything like Mechanize? Handling cookie state and such is something that is much more useful than the selector functionality of jQuery...which is great, but not any better than what Nokogiri offers.

https://github.com/sgentle/phantomjs-node is pretty good for most tasks.

Re: Using Node.js and JQuery to Crawl Public Tweets

#12

Post by dmazin » Mon, Aug 06, 2012, 3:29 AM UTC

blyxa wrote: post #2

why not use the twitter api?

It's rate-limited to 150 requests per hour and you can only pull the 3,200 most recent tweets for a user.

Re: Using Node.js and JQuery to Crawl Public Tweets

#13

Post by laughinghan » Mon, Aug 06, 2012, 5:33 AM UTC

danso wrote: post #10

Does Node have anything like Mechanize? Handling cookie state and such is something that is much more useful than the selector functionality of jQuery...which is great, but not any better than what Nokogiri offers.

http://zombie.labnotes.org/ is a library I've used with great success. The documentation in particular is cute.

I found PhantomJS unnecessarily convoluted for trivial tasks and was unable to figure how to do the nontrivial thing I was actually trying to do. The documentation in particular was unusable.

Re: Using Node.js and JQuery to Crawl Public Tweets

#14

Post by wavephorm » Mon, Aug 06, 2012, 7:27 AM UTC

Using JQuery server-side to process Twitter posts which are already in JSON format is just so dumb I can believe I read an entirely usless blog post about it.

Re: Using Node.js and JQuery to Crawl Public Tweets

#15

Post by wskinner » Mon, Aug 06, 2012, 9:27 AM UTC

I have also found node+jQuery an effective web crawling combination. In particular the cheerio library https://github.com/MatthewMueller/cheerio greatly simplifies data extraction. And as others have mentioned, the asynchronous nature of node is perfectly suited to crawling (as long as you take care not to accidentally DDOS the target site).