Using Node.js and JQuery to Crawl Public Tweets
1–10 of 15 posts
Re: Using Node.js and JQuery to Crawl Public Tweets
#2why not use the twitter api?
Re: Using Node.js and JQuery to Crawl Public Tweets
#3why not use the twitter api?
from the github page:
Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML.
Re: Using Node.js and JQuery to Crawl Public Tweets
#4From the docs:
'''
* Node.js [...]
* jQuery [...]
[...]
This approach has become my hammer when web scraping tasks come up.
'''
If all you have is a hammer, you may find yourself noticing that objects become more nail-like :).
Re: Using Node.js and JQuery to Crawl Public Tweets
#5why not use the twitter api?
api has rate limits
Re: Using Node.js and JQuery to Crawl Public Tweets
#6why not use the twitter api?
from the github page: Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML.
A better (and practical) example is scraping an internet forum (I've done it, partially)
Re: Using Node.js and JQuery to Crawl Public Tweets
#7Re: Using Node.js and JQuery to Crawl Public Tweets
#8If you really want to scrape pages, you should use something like https://github.com/chriso/node.io/ which batches things in jobs, helps with error handling, io, etc...
Re: Using Node.js and JQuery to Crawl Public Tweets
#9why not use the twitter api?
The API probably has a key that can be blocked.
Not that I'm advocating this, but a potential advantage of scraping, is it can be combined with tor or proxies, etc to get around limits.
Re: Using Node.js and JQuery to Crawl Public Tweets
#10Does Node have anything like Mechanize? Handling cookie state and such is something that is much more useful than the selector functionality of jQuery...which is great, but not any better than what Nokogiri offers.