Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Practical Intro to Data Science

blog.zipfianacademy.com

Post reply on HN
Print view View original

11–20 of 37 posts

Re: A Practical Intro to Data Science

#11

Post by minimize_me »

rm999 wrote: post #9

>While R is the de facto standard for performing statistical analysis, it has quite a high learning curve What? R has a ridiculously low learning curve. I remember literally the first time I used R I loaded up a dataset and had a histogram and qqplot within 5 minutes and 3 or 4 lines of code. Just figuring out what libraries I would need to do that in python (and installing them) would probably take me at least 30 mi…

Depends on the analysis lifecycle I think - I hate using Matlab for writing any sort of large experimental framework in, especially if it has to interact with other systems, but for quick and dirty experiments, plotting, or parallelizing totally independent for-loops eg. for a grid-search over some model hyper-parameters, it's great (the latter is literally changing "for" to "parfor". After you fork out for the Parallel Computing Toolbox of course.) But then a lot of code in research is written in the style of use-code-for-a-week->get a graph->curve is higher than their curve?->publication.

When I need to write something more substantive and reusable though, Python forever.

Re: A Practical Intro to Data Science

#12

Post by SkyMarshal »

danso wrote: post #7
darksaints wrote: post #4

Pretty decent list of links. However, I feel the importance of SQL has been completely downplayed...there are more hadoop-oriented links than there are SQL. Data retrieval and manipulation is where a data scientist will spend 95% of her time, and SQL is still more ubiquitous by far.

Dang, I wish I could find the link to this...an HP data scientist wrote a short essay (something like "Intro to Data Science") and said that the proper collection and cleaning of data is often seen as dirty grudge work that has to be done (by someone else, hopefully) before the real groundbreaking work can be done. However, the author said, this dirty grudge work is the real work. When I think about, in my data progr…

>Dang, I wish I could find the link to this...an HP data scientist wrote a short essay (something like "Intro to Data Science") and said that the proper collection and cleaning of data is often seen as dirty grudge work that has to be done (by someone else, hopefully) before the real groundbreaking work can be done. However, the author said, this dirty grudge work is the real work.

It's called data munging. Good short article on dataspora about it a while back:

http://www.dataspora.com/2009/05/sexy-data-geeks/

Re: A Practical Intro to Data Science

#13

Post by theaceae »

tomrod wrote: post #10
recuter wrote: post #5

Earlier quoted context omitted.

Well we might as well split it into Big Data and "Little Data". Little Data being: Have a basic grasp of Python and Javascript/D3.js for the pretty visualizations. That and basic statistics. The latter is probably the one developers (at least here) would spend most of their effort on. "Little Data" in itself can take you a long way.

What is D3.js?

A JavaScript data visualization library (d3js.org is linked to in the post)

A really friendly place to start understanding it is Scott Murray's tutorial: http://alignedleft.com/tutorials/d3/

Re: A Practical Intro to Data Science

#14

Post by ahi »

rm999 wrote: post #9

>While R is the de facto standard for performing statistical analysis, it has quite a high learning curve What? R has a ridiculously low learning curve. I remember literally the first time I used R I loaded up a dataset and had a histogram and qqplot within 5 minutes and 3 or 4 lines of code. Just figuring out what libraries I would need to do that in python (and installing them) would probably take me at least 30 mi…

Some things, maybe most things that a beginner wants to do are easy, but beyond the basics, R has a cliff for a learning curve. It's a complex language with some questionable design choices.

Re: A Practical Intro to Data Science

#15

Post by rm999 »

ahi wrote: post #14
rm999 wrote: post #9

>While R is the de facto standard for performing statistical analysis, it has quite a high learning curve What? R has a ridiculously low learning curve. I remember literally the first time I used R I loaded up a dataset and had a histogram and qqplot within 5 minutes and 3 or 4 lines of code. Just figuring out what libraries I would need to do that in python (and installing them) would probably take me at least 30 mi…

Some things, maybe most things that a beginner wants to do are easy, but beyond the basics, R has a cliff for a learning curve. It's a complex language with some questionable design choices.

I agree with you about questionable design choices, I really don't like developing in R. But if you have some data and you want to know more about that data (data analysis), R + a small handful of packages is the best free environment that I know of. It's not just the basics, this can extend into fairly complex operations on your data (not to mention producing very pretty visualizations).

Data analysis is one of the most important steps in data science, so I think it's worth keeping R around.

Re: A Practical Intro to Data Science

#16

Post by zissou »

Happy to finally see an intro to data science article that puts statistics as the #1 skill. While software engineering is also important, all the engineering in the world won't help you extract any meaningful insight from data without a solid foundation in probability and statistical theory.

Re: A Practical Intro to Data Science

#18

Post by clearspandex »

rm999 wrote: post #9

>While R is the de facto standard for performing statistical analysis, it has quite a high learning curve What? R has a ridiculously low learning curve. I remember literally the first time I used R I loaded up a dataset and had a histogram and qqplot within 5 minutes and 3 or 4 lines of code. Just figuring out what libraries I would need to do that in python (and installing them) would probably take me at least 30 mi…

Jonathan here, Co-Founder of Zipfian Academy. While it is easy to get up and running quite easily in R for simple analysis, it is a complex language that takes years to master. I recommend learning Python for the aspiring scientist because of its breadth of applicability. While R is probably better for statistical analysis than Python (every language has its specific domain where it shines), across the entire domain of tasks a data scientist must perform, I feel that Python provides the best aggregate utility.

Also, as the comments below highlight, actual statistical analysis is but a small part of the data pipeline. Python has great facilities for interacting with data stores/sources in addition to being a powerful tool to clean and munge data.

>When I think about, in my data programming related work, I'd say about 5% is doing analysis or executing statistical routines. And 95% of my time is spent on finding, cleaning, and properly normalizing data.

I hope the post doesn't downplay the importance of R to statistical analysis, it is a mature language with a great community surrounding it. The toolset of a data scientist is probably one of the most heterogeneous out there and necessitates learning and using many different abstractions.

For such a new (and hard to define) subject, I think dialogue is crucial to constructively advance the field. I would love to hear suggestions from the HN community about how to train the next generation of data scientist, what aspiring data scientists want to learn (or find difficult to learn), and how we can build a great data community.

Post reply on HN