Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to learn data science

dataquest.io

Post reply on HN
Print view View original

71–80 of 83 posts

Re: How to learn data science

#71

Post by jwuphysics »

probdist wrote: post #9
minimaxir wrote: post #6

The actual problem with learning "data science" is making inferences and conclusions which do not violate the laws of statistics . I've seen many submissions to Hacker News and Reddit's /r/dataisbeautiful subreddit where the author goes "look, the analysis supports my conclusion and the R^2 is high, therefore this is a good analysis!" without addressing the assumptions required for those results. Of course, not every…

The classic venn diagram of data science is still helpful: http://drewconway.com/zia/2013/3/26/the-data-science-venn-di... This article reads like a way to find yourself in the danger zone.

I've never seen this venn diagram before--thanks for bringing it up. I find that, as an academic (pursuing a Ph.D. in astrophysics) that plenty of traditional researchers are able to hack together code (many haven't ever taken a formal programming course; http://arxiv.org/abs/1507.03989) but many also misuse or can't interpret statistics (from personal experience). That puts us in the danger zone!

I think the key is to find the mathematics and statistics interesting because you want the [data] science to be meaningful. If that's a driving force, then you can learn math and statistics on your own (like the author did). Otherwise, yes--you will find yourself in the danger zone.

Re: How to learn data science

#72

Post by searine »

danenania wrote: post #69
searine wrote: post #54

Earlier quoted context omitted.

No I've just seen too many people spin their wheels on "analysis" that is not hypothesis driven. You got to start with questions to get answers, and the hard part of science isn't crunching data, it is asking the right question!

And how does the Right Question appear if not through exploration and manipulation of the data? Theory can obviously be very useful, but much of this stress on advanced statistics and phds is just a smokescreen for academics who suck at programming. If you can't program and manipulate data, statistics won't save you because you won't have the ability to dig deep enough to find valuable insights. On the other side, if…

>And how does the Right Question appear if not through exploration and manipulation of the data?

Questions don't magically come out of a data set. Doing so is called a fishing expedition and usually results in boring, descriptive results which have no impact.

To answer impactful questions, you must go into your data collection with the questions in mind. To understand what questions to ask, you need a trained, critical, and creative mind. That is something you don't get from pushing bits.

>If you can't program and manipulate data

Programming, and manipulating data is easy. Almost every new statistician these days can, and does do this routinely.

What's hard is the years of intuition about what is meaningful and what is noise.

I know. It's hard to hear, and career programmers most of all hate to hear it, but its the truth.

Re: How to learn data science

#73

Post by danenania »

searine wrote: post #72
danenania wrote: post #69

Earlier quoted context omitted.

And how does the Right Question appear if not through exploration and manipulation of the data? Theory can obviously be very useful, but much of this stress on advanced statistics and phds is just a smokescreen for academics who suck at programming. If you can't program and manipulate data, statistics won't save you because you won't have the ability to dig deep enough to find valuable insights. On the other side, if…

>And how does the Right Question appear if not through exploration and manipulation of the data? Questions don't magically come out of a data set. Doing so is called a fishing expedition and usually results in boring, descriptive results which have no impact. To answer impactful questions, you must go into your data collection with the questions in mind. To understand what questions to ask, you need a trained, critic…

Anyone I've ever heard say "programming is easy" is without fail a terrible programmer.

I'm not really sure how to respond to the idea that exploring a dataset isn't a useful way to help develop questions about it. It's only a "fishing expedition" if you have no idea what you're doing.

Re: How to learn data science

#75

Post by stared »

S4M wrote: post #60
stared wrote: post #37

Earlier quoted context omitted.

I would shift priorities to Python (from SQL). Unless one has "data scientist" title so to make "database engineer" look more fancy, then data comes in various shapes and forms. And most questions cannot be answered with a simple aggregation. For example, data I work on (I am a data scientist freelancer) is flat csv files, xls files, JSON files, some text files I need to parse, various SQL, MongoDB, things I am getti…

I would disagree with that advice. If you work as a data scientist in a company, you will likely have the logs of something stored in an SQL table (be it pure SQL database or something like hadoop hive) and you will have to answer (and ask) to questions like: "Do people convert more when they come from X or Y?", so you will have to do a couple of queries to get the conversion rates from people coming from X and Y. Th…

I work as a freelancer. And actually, I never ever processed logs.

Of course, sometimes I am given SQL access to server; but I never learnt SQL except for in action (i.e. things which I need right now).

And most of times I work with flat files. Even if they come from SQL they typically need a serious preprocessing before I can do a more adv analysis.

BTW: I have no problems with composing rather advanced queries. Just if SQL is a problem from someone (and, in case of doubt, it can't be Googled in no time) then I am curious how can get machine learning.

Re: How to learn data science

#76

Post by searine »

danenania wrote: post #73
searine wrote: post #72

Earlier quoted context omitted.

>And how does the Right Question appear if not through exploration and manipulation of the data? Questions don't magically come out of a data set. Doing so is called a fishing expedition and usually results in boring, descriptive results which have no impact. To answer impactful questions, you must go into your data collection with the questions in mind. To understand what questions to ask, you need a trained, critic…

Anyone I've ever heard say "programming is easy" is without fail a terrible programmer. I'm not really sure how to respond to the idea that exploring a dataset isn't a useful way to help develop questions about it. It's only a "fishing expedition" if you have no idea what you're doing.

>Anyone I've ever heard say "programming is easy" is without fail a terrible programmer.

Development of a worldclass application, is difficult because of the complexity built into a program of large scope.

Knowing enough programming to competently move a data set around, is easy. Hell you could do most of it with just bash.

>I'm not really sure how to respond to the idea that exploring a dataset isn't a useful way to help develop questions about it. It's only a "fishing expedition" if you have no idea what you're doing.

Well I've seen a lot of it, in both science and business. People who spend a lot of time and money to generate a large data set simply because they lack a question to ask. They expect meaningful answers to just tumble out of it like mana from heaven, and end up confused and dismayed when the answers aren't impactful.

Fishing expeditions are looked down upon because they can only describe the data you generated. That is minimally useful, and can be done without grabbing a huge sample.

Good science starts with a question, then puts data to work to create new insight by removing confounding factors through careful design.

Re: How to learn data science

#77

Post by mikedmiked »

stdbrouw wrote: post #27

Definitely agree that those long lists that tell you to first become awesome at combinatorics, linear algebra, then learn all about statistical inference (that is, not actual statistical procedures but the mathematical underpinnings of statistics that would enable you to construct and evaluate methods you invent yourself), then move on to stochastic optimization... those are really more about machismo than about actu…

+1 for DataTau, I didn't know about that.

http://www.datatau.com/

Check out this too: http://www.pyquantnews.com/

Re: How to learn data science

#79

Post by rjusher »

eitally wrote: post #57
rjusher wrote: post #51

Earlier quoted context omitted.

But how do you become a good data scientist, instead of a technical person, that knows how to apply an algorithm in Python/R. What I am trying to ask is how do you become good at setting your start point(formulate your hypotheses), communicating your insights and selecting which tools apply where, because if your are good at coding and have experience in things related to computer science you have the abilities to ha…

Practice. And education, but mostly practice. This is the kind of thing that is typically taught in formal educational settings (at least in engineering, which is my experience). As an example, I learned more about probability & statistics in 1) AP biology in high school, and 2) a "simulation systems" class in my industrial engineering master's curriculum. We spent much of the former class learning basic statistical…

Would you recommend any approach or I should go undust my high school and college books in the search for study material. Or is this too basic material.
Post reply on HN