Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

spectrum.ieee.org

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#41

Post by pessimizer »

mathgenius wrote: post #3

These two comments seem to contradict each other: "we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like." "...you get an output from the end of the layers, and you propagate a signal backwards through the layers to change all the parameters. It’s pretty clear the brain doesn’t do something like that. " So w…

As a weak epiphenominalist, I'd argue that all conscious thought is a backpropagation. Is it controversial that the brain is able to perceive its own output, or am I misunderstanding how backpropagation in neural networks is implemented?

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#42

Post by michaelochurch »

muraiki wrote: post #38
Daishiman wrote: post #24

Great interview. In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in statistics and algorithms to get anything of any value done. In my experience there are very few domains within Machine Learning where you don't need to be an expert in the field to yield useful conclusions out of the data. Even if you have a high-…

As someone who has tried out various MOOCs and entry level resources on machine learning, this is the same conclusion I came to. Beyond any sort of trivial example, I found I lacked the mathematical and statistical knowledge to not only interpret the results in a relatively unbiased and error-free way, but to know "what to do next." What scares me is that MOOCs are really pushing the data scientist field -- see Udaci…

Beyond any sort of trivial example, I found I lacked the mathematical and statistical knowledge to not only interpret the results in a relatively unbiased and error-free way, but to know "what to do next."

The popular MOOCs don't take you far enough to start doing serious machine learning, but you don't need a PhD to be ready to solve those problems.

It takes work. Lots of work. Re-learn linear algebra until you know why "eigenvectors" are so important. Know what the most important matrix factorizations (LU, QR, SVD, Eigen, Cholesky) do. Read the papers until the math becomes "no big deal". Pick up a probability textbook and read the whole thing; also, get a working knowledge of real analysis. It won't happen quickly.

The PhD is some classes, plus 3-7 years of focused work. Some of that's compressible and unnecessary to becoming a data scientist. Some of it isn't. The Coursera courses are great for getting you started; they're entry-level college courses, and if you read the papers and the seminal textbooks (e.g. Elements of Statistical Learning by Hastie et al) you can get into the intermediate territory in a couple years or so. It's not easy, but it can definitely be done. Getting to the expert level, I think, just requires real-world experience on real-world problems... but, one hopes, you can start attacking such problems once you're at the intermediate level.

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#43

Post by michaelochurch »

Daishiman wrote: post #24

Great interview. In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in statistics and algorithms to get anything of any value done. In my experience there are very few domains within Machine Learning where you don't need to be an expert in the field to yield useful conclusions out of the data. Even if you have a high-…

In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in statistics and algorithms to get anything of any value done.

The fact that you think it takes a PhD to be a decent data scientist indicates that you're out of your depth on this one.

You don't need a PhD to get useful work done in these fields. You need to work hard and tackle difficult math. It takes years, but it can be done if you have the talent and drive. A prestigious (top-10) PhD certainly makes your life easier in getting the top jobs, but it doesn't really make you more (or less) able to fulfill them. I don't have a PhD and can do what 95+ percent of "PhD Data Scientists" do for work.

PhD is (a) focused work on a specific, usually narrow problem, and (b) years of self-study that required to know enough to attack said problem. For real-world data science, (a) only matters in the ~1% chance of overlap between your dissertation and the needs of your employer, and (b) doesn't require five years in an academic institution (although it probably does require about that much time, if you study on your own, since you're likely to be doing much of the work on your own time).

The PhD is a valuable experience and I don't mean to denigrate it. I often wish I had gotten one, in my 20s, instead of becoming a world-class expert on software office politics and "only" an intermediate-plus Haskell/Clojure/machine-learning guy. The PhD is a great experience for many people, but I don't think it belongs on a pedestal.

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#44

Post by [deleted] »

michaelochurch wrote: post #42
muraiki wrote: post #38

Earlier quoted context omitted.

As someone who has tried out various MOOCs and entry level resources on machine learning, this is the same conclusion I came to. Beyond any sort of trivial example, I found I lacked the mathematical and statistical knowledge to not only interpret the results in a relatively unbiased and error-free way, but to know "what to do next." What scares me is that MOOCs are really pushing the data scientist field -- see Udaci…

Beyond any sort of trivial example, I found I lacked the mathematical and statistical knowledge to not only interpret the results in a relatively unbiased and error-free way, but to know "what to do next." The popular MOOCs don't take you far enough to start doing serious machine learning, but you don't need a PhD to be ready to solve those problems. It takes work. Lots of work. Re-learn linear algebra until you know…

[deleted]

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#45

Post by sharmi »

michaelochurch wrote: post #42
muraiki wrote: post #38

Earlier quoted context omitted.

As someone who has tried out various MOOCs and entry level resources on machine learning, this is the same conclusion I came to. Beyond any sort of trivial example, I found I lacked the mathematical and statistical knowledge to not only interpret the results in a relatively unbiased and error-free way, but to know "what to do next." What scares me is that MOOCs are really pushing the data scientist field -- see Udaci…

Beyond any sort of trivial example, I found I lacked the mathematical and statistical knowledge to not only interpret the results in a relatively unbiased and error-free way, but to know "what to do next." The popular MOOCs don't take you far enough to start doing serious machine learning, but you don't need a PhD to be ready to solve those problems. It takes work. Lots of work. Re-learn linear algebra until you know…

Thanks muraiki and michaelochurch. This is a similar frustration I faced. The moocs often seem to teach you just enough that is similar to formula substitution, everything starts to crumble when you depart towards data that's significantly different. So I have begun from the bottom starting with MIT's Linear Algebra and Harvard's Statistics 110. Your comments have validated my journey though this is going to be a long one.

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#46

Post by asdavis »

jimduk wrote: post #31
sherjilozair wrote: post #23

Earlier quoted context omitted.

This is a reply to multiple sibling comments. There is actually recent work which shows that deep learning methods can also work WITHOUT any reverse signals: http://s.yosinski.com/dan_cownden_presentation.pdf

Interesting paper. Any more details on the architecture of the feedback connections. Also I can't tell from the paper where and how weights are being updated e..g what does "train" mean in this context

I believe instead of multiplying the delta by W^t to backpropagate the error from layer l to l-1, you multiply it by a random projection B. It's hard to dig deeper because there doesn't appear to be any other information on it except here: http://isis-innovation.com/licence-details/accelerating-mach...

As an aside, are they really trying to patent a slight twist on backpropagation? That seems pretty counter-productive to me.

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#47

Post by asdavis »

sherjilozair wrote: post #23
mathgenius wrote: post #3

These two comments seem to contradict each other: "we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like." "...you get an output from the end of the layers, and you propagate a signal backwards through the layers to change all the parameters. It’s pretty clear the brain doesn’t do something like that. " So w…

This is a reply to multiple sibling comments. There is actually recent work which shows that deep learning methods can also work WITHOUT any reverse signals: http://s.yosinski.com/dan_cownden_presentation.pdf

It looks like there are still reverse signals (eg., deltas), but they are multiplied by a random matrix B instead of the usual transpose of the weight matrix for that layer. Am I misunderstanding?

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#48

Post by dnautics »

pessimizer wrote: post #41
mathgenius wrote: post #3

These two comments seem to contradict each other: "we have no idea how neurons are storing information, how they are computing, what the rules are, what the algorithms are, what the representations are, and the like." "...you get an output from the end of the layers, and you propagate a signal backwards through the layers to change all the parameters. It’s pretty clear the brain doesn’t do something like that. " So w…

As a weak epiphenominalist, I'd argue that all conscious thought is a backpropagation. Is it controversial that the brain is able to perceive its own output, or am I misunderstanding how backpropagation in neural networks is implemented?

that's not what backpropagation is. Backpropagation is best thought of as a 'cheat' (algorithmic simplification) that allows you to calculate the derivative of a feed-forward neural net. You would need to calculate the derivative of a neural net in general to optimize relative to some cost function, you could use, for example, gradient descent, but that is computationally costly.

For some neural nets, you still have a gradient, but the concept of back or forward propagation is not defineable. Based on the topology and structure of biological neural nets, what would you think is the case?

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#49

Post by Daishiman »

michaelochurch wrote: post #43
Daishiman wrote: post #24

Great interview. In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in statistics and algorithms to get anything of any value done. In my experience there are very few domains within Machine Learning where you don't need to be an expert in the field to yield useful conclusions out of the data. Even if you have a high-…

In my experience it's amazing just how many people are talking about "Big Data" and just how exactly none of those are the ones with the necessary PhDs in statistics and algorithms to get anything of any value done. The fact that you think it takes a PhD to be a decent data scientist indicates that you're out of your depth on this one. You don't need a PhD to get useful work done in these fields. You need to work har…

I'm not talking strictly about a PhD, but about PhD level work, some of which can't really be done on your own unless you're a supremely talented individual.

You can look at some of the modern ML algorithms and see what I mean; many people that I know have worked with Latent Dirichlet Allocation, but they have no idea how the model works, and there's no way they could extend to work online or under certain performance or storage constraints without havin spent months and years working on that problem.

That's not a realistic expectation for anyone in the field. Yes, the algorithms you find in Weka and other ML toolkits are useful, but the actual "Big Data" problems have their own performance and algorithmic constraints that are far, far beyond dedicated self-learners.

Re: Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Others

#50

Post by scottlocklin »

jostmey wrote: post #17

I am glad he pointed out that most artificial neural networks bear only a superficial resemblance to our own biological ones. But I think he failed to appreciate the power behind Boltzmann Machines - a type of neural network designed to create a generative model of a dataset. Personally, I find the resemblance between these neural networks and the real ones a little uncanny. And very few people seem to realize that t…

I've only started looking at RBMs recently, but ... what are you talking about? Biological neural networks use spikes. RBMs certainly don't. RBMs look more like HMMs to me than like biological neurons. Don't take this as me saying, "you're wrong" -I'm curious if there is another way to think about RBMs (aka, "papers please" -so I have a deeper understanding when I do my own implementation of RBMs).
Post reply on HN