Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R for Data Science

r4ds.had.co.nz

Post reply on HN
Print view View original

31–40 of 76 posts

Re: R for Data Science

#31

Post by vegabook »

Kevin_S wrote: post #18

Seems interesting. Quick question: Some background on myself first. I am a financial consultant (only 1 year since graduating) and am planning to do a PhD in Accounting in the next 3 years. Currently working through the GMAT, but once that is complete, I will find myself with 2 or so years to do things that will help prepare me for research. One thing I have considered is taking a course/reading books on data science…

Personally I work in macro and fixed income market analysis (strategist), and I can heartily recommend R as your first language. Indeed, coming from a CS background, I first applied Python to many problems, and resisted R which was not a "grown up" programming language, in my opinion (some would make the same accusation on Python). However I dipped my toe in the water one day because R had a Bloomberg terminal add in and Python did not (at the time), and after about a month of uphill learning curve the eureka moments started materializing thick and fast. I cannot recommend R enough, as a problem exploration language. It just beats Python hands down when it comes to grabbing some (usually dirty) data, mangling it around, cleaning it, and then install.package'ing a bunch of potentially useful libraries which allow you to do everything you could possibly imagine to a small to medium sized data set. And crucially, static graphing. Nothing else comes close for this use case.

Now...caveats. R is not a production programming language. If you find yourself creating something truly useful for many users, that requires robust programming language structures such as threading, proper memory management, server-capability, or indeed, speed, R is going to become frustrating. Yes a whole bunch of people will tell you "it's possible, I do it, etc", but that is not its sweet spot. Also, if your data set is bigger than 2-3 gig or so, you're going to start hitting R's memory management wall. It's slow. You'll then be better off with Python, C, or indeed, Scala, or possibly, Apache Spark. The common thing about these caveats, however, is that they're definitely second order problems, later in your career life cycle, than the excellent mainstream data science tool which is R for people who have outgrown Excel, but are not full fledged computer scientists, and who want to get (lots of) stuff, done.

(by the way, pre-empting comments. Yes Pandas is great, but no it's not quite R).

Re: R for Data Science

#32

Post by Kevin_S »

vegabook wrote: post #31
Kevin_S wrote: post #18

Seems interesting. Quick question: Some background on myself first. I am a financial consultant (only 1 year since graduating) and am planning to do a PhD in Accounting in the next 3 years. Currently working through the GMAT, but once that is complete, I will find myself with 2 or so years to do things that will help prepare me for research. One thing I have considered is taking a course/reading books on data science…

Personally I work in macro and fixed income market analysis (strategist), and I can heartily recommend R as your first language. Indeed, coming from a CS background, I first applied Python to many problems, and resisted R which was not a "grown up" programming language, in my opinion (some would make the same accusation on Python). However I dipped my toe in the water one day because R had a Bloomberg terminal add in…

Thanks for the replies everyone. I'll definitely save this link and pick up the book when it comes out!

Re: R for Data Science

#33

Post by hadley »

geebee wrote: post #30
hadley wrote: post #2

I'm the author, and I'm happy to answer any questions. The book should be in print by (hopefully) the end of this year, or definitely by Jan 2017. The content will not change significantly, but there is will be minor fixes and a lot of proof reading.

Great book, I'm getting a lot out of the site and I'm looking forward to the release. Thanks! I understand there is always one more library or topic that could be included... .. but with that acknowledged, what do you think of sqldf as an alternative to dplyr? You mention that dplyr is a bit easier (within the context of being specialized for data analysis). I'd have trouble weighting in because I don't use R all tha…

If you don't already know SQL or dplyr, I think you would find dplyr significantly easier to learn. Some people who do know SQL well have commented that they too find dplyr easier. I think this is because the scope of dplyr is much smaller than SQL and it is designed specifically to facilitate data analysis.

Re: R for Data Science

#34

Post by javitury »

Kevin_S wrote: post #18

Seems interesting. Quick question: Some background on myself first. I am a financial consultant (only 1 year since graduating) and am planning to do a PhD in Accounting in the next 3 years. Currently working through the GMAT, but once that is complete, I will find myself with 2 or so years to do things that will help prepare me for research. One thing I have considered is taking a course/reading books on data science…

I'm coursing a research/PhD access master degree and all I can tell you R is one of the most popular tools. If it is used in your university go for it.

It's not just the features of the R programming language. It is also free, open source and can be integrated easily with many external tools. For example you can make a paper using Markdown(RMarkdown) and convert it to ->TEX->PDF. Or you can use Rmarkdown to make a presentation converting it to ->TEX->beamer slides. Or just output a html document/webpage. It is also well integrated with online databases and resources.

While it's true that it can be a bit less intuitive or not as visualization focused as compared to some other tools, it is just as powerful under the hood.

Other open source tools like python are good for programming but when it comes to data analysis they can be a little bit cumbersome. For example, python is quite object oriented and for straightforward purposes with low reusability the amount of code needed can be large.

There are other tools like Matlab, Mathematica, SPSS... Matlab is good at visualization and Mathematica has really nice features aimed to improve understanding. However these are closed source and cost a lot of money to you or the university.

Re: R for Data Science

#35

Post by peatmoss »

geebee wrote: post #30
hadley wrote: post #2

I'm the author, and I'm happy to answer any questions. The book should be in print by (hopefully) the end of this year, or definitely by Jan 2017. The content will not change significantly, but there is will be minor fixes and a lot of proof reading.

Great book, I'm getting a lot out of the site and I'm looking forward to the release. Thanks! I understand there is always one more library or topic that could be included... .. but with that acknowledged, what do you think of sqldf as an alternative to dplyr? You mention that dplyr is a bit easier (within the context of being specialized for data analysis). I'd have trouble weighting in because I don't use R all tha…

I have a related technical question. Why couldn't something highly embeddable like SQLite be the default underlying implementation for a data frame in something like Python or R? It seems like Pandas and R data frames have a great deal redundant functionality.

SQLite seems like it has the guts to be the standard libdataframe.c for R, Python, Julia, etc. As a side benefit it already has a super consistent API (a.k.a. SQL).

Re: R for Data Science

#36

Post by hadley »

dreww2 wrote: post #27

Hadley, can you share a bit more about your plans for modelr and what need(s) the package will be designed to solve? Congrats on your book btw, I've been reading it for a few weeks and it's quite simply excellent.

I don't think modelr is going to change significantly in the future. It solved a pressing problem (fitting models as part of a pipeline) so I could teach modelling using the same interface as everything else in the book.

However, the modelling infrastructure in R is generally showing it's age, and thinking about how to make modelling easier is something that I will be working on in the coming months.

Re: R for Data Science

#37

Post by hadley »

peatmoss wrote: post #35
geebee wrote: post #30

Earlier quoted context omitted.

Great book, I'm getting a lot out of the site and I'm looking forward to the release. Thanks! I understand there is always one more library or topic that could be included... .. but with that acknowledged, what do you think of sqldf as an alternative to dplyr? You mention that dplyr is a bit easier (within the context of being specialized for data analysis). I'd have trouble weighting in because I don't use R all tha…

I have a related technical question. Why couldn't something highly embeddable like SQLite be the default underlying implementation for a data frame in something like Python or R? It seems like Pandas and R data frames have a great deal redundant functionality. SQLite seems like it has the guts to be the standard libdataframe.c for R, Python, Julia, etc. As a side benefit it already has a super consistent API (a.k.a.…

Because it's designed to support typically relational db workloads (i.e. Lots of changes) not data analysis workloads. Dataframes in R, pandas etc, are column oriented, which leads to better trade offs for analysis.

Also SQL is a substantially inferior API for data analysis. (Not because it's a bad language, but again because that's not what it's designed for)

Re: R for Data Science

#38

Post by peatmoss »

hadley wrote: post #37
peatmoss wrote: post #35

Earlier quoted context omitted.

I have a related technical question. Why couldn't something highly embeddable like SQLite be the default underlying implementation for a data frame in something like Python or R? It seems like Pandas and R data frames have a great deal redundant functionality. SQLite seems like it has the guts to be the standard libdataframe.c for R, Python, Julia, etc. As a side benefit it already has a super consistent API (a.k.a.…

Because it's designed to support typically relational db workloads (i.e. Lots of changes) not data analysis workloads. Dataframes in R, pandas etc, are column oriented, which leads to better trade offs for analysis. Also SQL is a substantially inferior API for data analysis. (Not because it's a bad language, but again because that's not what it's designed for)

Thank you for the reply--I immediately started Googling for more info about column-oriented data stores to see if there was something analogous to SQLite in this space. It looks like there's an embedded MoneDBLite package for R now that I'll need to check out.

Re: R for Data Science

#39

Post by geebee »

hadley wrote: post #37
peatmoss wrote: post #35

Earlier quoted context omitted.

I have a related technical question. Why couldn't something highly embeddable like SQLite be the default underlying implementation for a data frame in something like Python or R? It seems like Pandas and R data frames have a great deal redundant functionality. SQLite seems like it has the guts to be the standard libdataframe.c for R, Python, Julia, etc. As a side benefit it already has a super consistent API (a.k.a.…

Because it's designed to support typically relational db workloads (i.e. Lots of changes) not data analysis workloads. Dataframes in R, pandas etc, are column oriented, which leads to better trade offs for analysis. Also SQL is a substantially inferior API for data analysis. (Not because it's a bad language, but again because that's not what it's designed for)

I agree completely that SQL is not the language for the kind of data analysis you're discussing in this book - to me, the question is whether it's useful to do querying and filtering through SQL and data analysis through python and R on the resulting datasets. I think pretty much everything you've written here would be continue useful if you used sqldf to generate data frames in R, but I don't know R well enough to be sure of that.

Because pandasql returns a data frame from a data frame (not sure if this is the case with R), I find it relatively easy to do data things with sql and data analysis with python. However, that's not a huge surprise since I've been using SQL for a while but don't know the pandas or R data frame syntax especially well.

I'm not sure why sqlite was chosen - could it have to do with the in-memory nature of dataframes? So far, my use of sql with data frames has been pretty generic, so I haven't bumped up any implementation specific SQL issues.

Re: R for Data Science

#40

Post by dreww2 »

hadley wrote: post #36
dreww2 wrote: post #27

Hadley, can you share a bit more about your plans for modelr and what need(s) the package will be designed to solve? Congrats on your book btw, I've been reading it for a few weeks and it's quite simply excellent.

I don't think modelr is going to change significantly in the future. It solved a pressing problem (fitting models as part of a pipeline) so I could teach modelling using the same interface as everything else in the book. However, the modelling infrastructure in R is generally showing it's age, and thinking about how to make modelling easier is something that I will be working on in the coming months.

Sounds great. Something like a tidyverse version of the data pipeline capabilities of scikit-learn would be awesome.
Post reply on HN