Live data from Hacker News

How to learn data science

dataquest.io

61–70 of 83 posts

Re: How to learn data science

#61
post #59

Here are some topics. Are they considered relevant to data science ? Matrix row rank and column rank are equal. In matrix theory, the polar decomposition. Each Hermitian matrix has an orthogonal basis of eigenvectors. Weak law of large numbers. Strong law of large numbers. The Radon-Nikodym theorem and conditional expectation. Sample mean and variance are sufficient statistics for independent, identically distributed…

The topics you mention are maths or applied maths topics. "Data Science" is a bubbly term that roughly means "take that big dump of data and give me some advice on how to make more money", so your list, very sadly, has little relevance with it.

Re: How to learn data science

#62
post #26

Earlier quoted context omitted.

Roughly 80% of data scientists I know have PhD in something very math heavy. Rest have masters degrees. There are programmers who can assist them doing the grunt work but it's just basic programming to assist analysts to crunch data. If you want to do data science for real: 1. Get Masters of PhD from statistics, computer science, economics, physics or some other heavy field and specialize data analysis in that field.…

No. A phd in statistics or economics means almost nothing at this point. Even if it did, truly, signal mastery of the content, which it doesn't anymore, it would signal to most people who do this kind of work that you're way overqualified while simultaneously being totally ignorant of the day-to-day work of actual data scientists. If you want to be a useful data scientist, do a lot of work with data. If you have stro…

There absolutely are problems that require a more rigorous mathematical training than you get from undergraduate courses or day-to-day experience. Most data scientists and companies may not be tackling these problems, but they certainly exist.

Just having a PhD will open doors for you that would otherwise be shut. But before pursuing that degree, you should be confident that you enjoy working in the field and want to devote your career to it. Also, you have to be prepared to work hard, not just to get the degree, but then to land a job where you'll put that experience to use. Otherwise, you'll be sharing a cubicle with DataWorker and feeling like a fool.

That said, if you don't know whether you need a PhD, that means you probably don't know what kinds of problem you want to work on. And in that case, there's a good chance you'll end up working on a problem that only interests your advisor and nobody else (most PhD advisors have more students than they have good problems to work on). In that case, I wouldn't recommend it.

Re: How to learn data science

#63
post #59

Here are some topics. Are they considered relevant to data science ? Matrix row rank and column rank are equal. In matrix theory, the polar decomposition. Each Hermitian matrix has an orthogonal basis of eigenvectors. Weak law of large numbers. Strong law of large numbers. The Radon-Nikodym theorem and conditional expectation. Sample mean and variance are sufficient statistics for independent, identically distributed…

I've seen in other threads you recommended Neveu's book to cover some probability theory topics. Care to explain whether Halmos & Rudin would be sufficient pre-requisites?

Re: How to learn data science

#64
post #63
post #59

Here are some topics. Are they considered relevant to data science ? Matrix row rank and column rank are equal. In matrix theory, the polar decomposition. Each Hermitian matrix has an orthogonal basis of eigenvectors. Weak law of large numbers. Strong law of large numbers. The Radon-Nikodym theorem and conditional expectation. Sample mean and variance are sufficient statistics for independent, identically distributed…

I've seen in other threads you recommended Neveu's book to cover some probability theory topics. Care to explain whether Halmos & Rudin would be sufficient pre-requisites?

Halmos Measure Theory is a good prerequisite to Neveu. Rudin, Principles is a bit too little. Instead, the first half, the real half of Rudin's Real and Complex Analysis is a good prerequisite. So, is Royden's Real Analysis.

Neveu is elegant beyond belief, but Breiman, Probability, the SIAM book, available in paperback, is darned good, usually easier than Neveu, less elegant, closer to applications, and without some of the special Tulcea material in the back of Neveu. K. L. Chung also has a good, comparable book. Even if want Neveu to be your main probability book, which is fine, likely you should have alternative treatments.

Of course, there is Loeve, Probability -- written in English but somehow sounding like French. It has a lot, a little too much, but I liked the topics I studied in it. It turns out, Neveu and Breiman were both Loeve students.

Halmos, Measure Theory, is darned fun to read: It has the three series theorem and a famous exercise on regular conditional probabilities.

I learned the stuff from a course by A. Karr, a star student of E. Cinlar. Karr's course was the best course of any kind I ever took in school. Powerful material, beautifully presented, each day it was a shame to erase the board.

The exercises in Neveu are usually harder than the ones in Halmos, Breiman, and Chung.

Neveu makes probability a crown jewel of civilization.

The summer after Karr's course, I sat in the library for six weeks and walked out with a 50 page manuscript that was all the research and the first draft of my dissertation. Net, probability at the level of Neveu is darned powerful stuff, makes a lot in research, and research for applications, really easy -- that is, you really know just what the heck you are doing and can knock off new results having fun sitting in bed next to your wife while she watches TV (warning -- not gender neutral!).

What I've outlined is sometimes just called graduate probability. The biggest difference is that the whole subject makes daily use of measure theory.

I don't know how much you need in probability before starting on graduate probability. In my case, graduate probability was my first serious study of probability, and I never felt that I was not prepared.

But in my career I'd done a lot of practical work in both probability and statistics -- e.g., multivariate statistics, hypothesis testing, stochastic processes, digital filtering, the fast Fourier transform, beam forming (a case of antenna theory), power spectral estimation (US Navy sonar type stuff), how to get the central limit theorem out of digital filtering, and more, random number generation, etc. That work was plenty of intuitive background for graduate probability.

But in much of that work I was struggling due to what, really, at that level, is commonly weak basic knowledge of probability. So, after those struggles, seeing graduate probability be all clean and powerful was great.

I can't advise on just how much elementary probability you might need to have enough intuition to be comfortable with graduate probability. I will say, you do need both the intuitive experience and also the solid math.

I feel sorry for people who work in prob/stat without a background in grad prob: The elementary stuff is too often just confused from poor understanding from a poor background.

The sources I mentioned above were really the first sources from which I did any real study. Net, the elementary material of prob/stat is really too simple to be taken very seriously. So, for your first serious effort, just go for graduate probability from the sources above.

The Neveu, etc., material is much of the foundation for the secret sauce of my startup.

Re: How to learn data science

#65
post #61
post #59

Here are some topics. Are they considered relevant to data science ? Matrix row rank and column rank are equal. In matrix theory, the polar decomposition. Each Hermitian matrix has an orthogonal basis of eigenvectors. Weak law of large numbers. Strong law of large numbers. The Radon-Nikodym theorem and conditional expectation. Sample mean and variance are sufficient statistics for independent, identically distributed…

The topics you mention are maths or applied maths topics. "Data Science" is a bubbly term that roughly means "take that big dump of data and give me some advice on how to make more money", so your list, very sadly, has little relevance with it.

Most of those topics I listed are supposed to be good at taking data and saying how to "make more money"!

Re: How to learn data science

#66
post #64
post #63

Earlier quoted context omitted.

I've seen in other threads you recommended Neveu's book to cover some probability theory topics. Care to explain whether Halmos & Rudin would be sufficient pre-requisites?

Halmos Measure Theory is a good prerequisite to Neveu. Rudin, Principles is a bit too little. Instead, the first half, the real half of Rudin's Real and Complex Analysis is a good prerequisite. So, is Royden's Real Analysis . Neveu is elegant beyond belief, but Breiman, Probability , the SIAM book, available in paperback, is darned good, usually easier than Neveu, less elegant, closer to applications, and without som…

Thanks for the insights. Chung seems quite doable at my current level. I skimmed through it sometime ago. I borrowed a copy of Neveu and it seemed a bit harder.

Care to share other references you like. Real & complex analysis and algebra, in particular, are most welcome.

Re: How to learn data science

#67
post #59

Here are some topics. Are they considered relevant to data science ? Matrix row rank and column rank are equal. In matrix theory, the polar decomposition. Each Hermitian matrix has an orthogonal basis of eigenvectors. Weak law of large numbers. Strong law of large numbers. The Radon-Nikodym theorem and conditional expectation. Sample mean and variance are sufficient statistics for independent, identically distributed…

I've been doing data science for a while now, and for me personally:

Not really. The SVD is much more important. No. Yes. Yes. No (R-N) yes (CE). Yes. Yes. Yes. Personally, no. Only in the usage of MCMC. Yes. Yes. No. Of course. All the time. Yes. Yes. The most I'll do is remember to use the sample standard deviation. No. Yes. No. Yes. Yes. Yes. No. No. Yes. I just use a solver. See above. See above. Of course. Yes. Yes. Not privileged w/r/t/ other bases. Of course. I've never needed it. Ditto. As another tool in the toolbox. They would not be my first or second choice. Yes. No. No. Yes. No. Yes. Yes. Yes. No.

Re: How to learn data science

#68
post #55

Anyone interested in data science should first study cognitive psychology. The CIA has a manual on the psychology of intelligence analysis that is a must read for anyone pursuing any analytical job. If you dont understand how your mind sees, processes, retains and recalls data...how can you possibly analyze it accurately?

You have a link to where to obtain said manual?

https://www.cia.gov/library/center-for-the-study-of-intellig...

Re: How to learn data science

#69
post #54

Earlier quoted context omitted.

Sounds like you've also spent years molding professional disdain for everyone who's not a Real Scientist.

No I've just seen too many people spin their wheels on "analysis" that is not hypothesis driven. You got to start with questions to get answers, and the hard part of science isn't crunching data, it is asking the right question!

And how does the Right Question appear if not through exploration and manipulation of the data?

Theory can obviously be very useful, but much of this stress on advanced statistics and phds is just a smokescreen for academics who suck at programming.

If you can't program and manipulate data, statistics won't save you because you won't have the ability to dig deep enough to find valuable insights. On the other side, if you know how to slice and dice data quickly and reliably, you can learn a huge amount by applying only the simplest statistical techniques. Generally the simple techniques are better anyway because they make mistakes less likely and your findings are easier to communicate.

Re: How to learn data science

#70

Earlier quoted context omitted.

> Good data science is not based on collecting large amounts of data passively and then mining it mindlessly. You need to ask right questions and design data collection and modeling process based on those questions. This resonates. That is, picking and designing features. Also understand dependent variables and knowing how to test for that, which is the biggest mistakes leading to flawed conclusions I see from the 'g…

What do you mean by testing for dependent variables?

Maybe something to do with instrumental variables? https://en.wikipedia.org/wiki/Instrumental_variable
Post reply on HN