Webscraping with Rvest
programmingr.com
Webscraping with Rvest
1–10 of 15 posts
Re: Webscraping with Rvest
#2The reason why so many people were mixing Python code with R was specifically for these sort of task. Web scraping in R has really caused me to not touch another tool outside of R for a few years now and it is great.
Well done Hadley Wickham being inspired by libraries like Beautiful Soup and bringing a great tool to R.
Re: Webscraping with Rvest
#3Rvest works fine with tabular data. If, however, you are working with data outside of Wikipedia, you will find that website data is very rarely available in a and is instead part of a hierarchical tree, which is a pain to process/clean in R.
In such cases, working with Python/BeautifulSoup4 and importing the clean and normalized data into R will save frustration over time, even offsetting the overhead of using two languages.
Re: Webscraping with Rvest
#4It really looks as easy as it can get. The good part of R is that many R packages are designed in a similar way (highly specialized methods, doing a good job). Combining that with %>% makes you really efficient.
Re: Webscraping with Rvest
#5Rvest works fine with tabular data. If, however, you are working with data outside of Wikipedia, you will find that website data is very rarely available in a and is instead part of a hierarchical tree, which is a pain to process/clean in R. In such cases, working with Python/BeautifulSoup4 and importing the clean and normalized data into R will save frustration over time, even offsetting the overhead of using two la…
I will work with any data, as soon as it is easily retrieved with some css selector. Otherwise you would have problems using any web scraping tool.
Re: Webscraping with Rvest
#6I've written a few blog posts where I used Rvest to get data and R's great visualization tools to visualize it. R has a ton of issues as a platform and language but this is a fantastic package and it has a great ecosystem for small data (the majority of data).
Re: Webscraping with Rvest
#7all of these web scraping frameworks, doesn't it tell you that the web needs more wide spread semantic markup?
YCombinator ...
Something like that?
I know what everyone will say, it is so terse and convoluted, but maybe something like
YCombinator
...Seems like a lot of work though...maybe I take that back.
Re: Webscraping with Rvest
#8Rvest works fine with tabular data. If, however, you are working with data outside of Wikipedia, you will find that website data is very rarely available in a and is instead part of a hierarchical tree, which is a pain to process/clean in R. In such cases, working with Python/BeautifulSoup4 and importing the clean and normalized data into R will save frustration over time, even offsetting the overhead of using two la…
I will work with any data, as soon as it is easily retrieved with some css selector. Otherwise you would have problems using any web scraping tool.
JSON is pretty easy to unpack, if you can figure out the call back that gets the data.
Re: Webscraping with Rvest
#9Earlier quoted context omitted.
I will work with any data, as soon as it is easily retrieved with some css selector. Otherwise you would have problems using any web scraping tool.
JSON is pretty easy to unpack, if you can figure out the call back that gets the data.
The primary use case for web scraping tools like Rvest is for data that doesn't have a JSON endpoint and everything is rendered serverside, or is a static web page.
Re: Webscraping with Rvest
#10all of these web scraping frameworks, doesn't it tell you that the web needs more wide spread semantic markup? YCombinator ... Something like that? I know what everyone will say, it is so terse and convoluted, but maybe something like YCombinator ... Seems like a lot of work though...maybe I take that back.
If websites wanted to make their data accessible, they would create APIs. Data on websites is inaccessible for a reason.