I've had great success with Ruby/Mechanize for regular html scraping and phantomjs for dynamic page scraping.
Webscraping with Rvest
11–15 of 15 posts
Re: Webscraping with Rvest
#12Rvest is for webscrapping newbs. A more seasoned R person would still use PhantomJS and RSelenium as it actually collects all the page's information but Rvest only collections a portion of it. Try it on washingtonpost.com and you will see.
Re: Webscraping with Rvest
#13Rvest works fine with tabular data. If, however, you are working with data outside of Wikipedia, you will find that website data is very rarely available in a and is instead part of a hierarchical tree, which is a pain to process/clean in R. In such cases, working with Python/BeautifulSoup4 and importing the clean and normalized data into R will save frustration over time, even offsetting the overhead of using two la…
> In such cases, working with Python/BeautifulSoup4
BUT Rvest is a BeautifulSoup inspired library and works pretty much the same way?
Re: Webscraping with Rvest
#14Rvest is for webscrapping newbs. A more seasoned R person would still use PhantomJS and RSelenium as it actually collects all the page's information but Rvest only collections a portion of it. Try it on washingtonpost.com and you will see.
> Rvest is for webscrapping newbs
down voted for calling people newbs. Also it always depends on what tool works best.
Re: Webscraping with Rvest
#15This seems really an intuitive way of getting the tables. What would be the most similar library in python for those cases where R isn't available in the system (with the permissions in some labs machines, unfortunately it takes weeks-forever to get R installed)?