Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding the History of Deep Learning

blog.floydhub.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: Coding the History of Deep Learning

#11

Post by letlambda »

dpcx wrote: post #6

This seems like a great introduction to the history. I have a problem with it, though. In the first example, the method compute_error_for_line_given_points is called with values 1, 2, [[3,6],[6,9],[12,18]]. Where did those values come from? Later in that same example, there is an "Error = 4^2 + (-1)^2 + 6^2". Where did those values come from? Later, there's another form: "Error = x^5 - 2x^3 -2" What about these? Ther…

>Am I missing something fundamental here? Yeah, these aren't magic formula, they are just examples.

>In the first example, the method compute_error_for_line_given_points is called with values 1, 2, [[3,6],[6,9],[12,18]]. Where did those values come from?

It's an example. The first two arguments define a line y = 2x + 1, the pairs are (x,y) points being used to compute the error.

"To play with this, let’s assume that the error function is Error=x^5−2x^3−2"

This is just an example of a function used as exposition to talk about derivatives.

It isn't even an error function though. An error function has to be a function of at least two variables.

Re: Coding the History of Deep Learning

#12

Post by terrabytes »

ska wrote: post #3
terrabytes wrote: post #2

Spot on. I struggled with the mainstream deep learning/machine learning MOOCs. I felt like they were to math heavy. However, I'm struggling on how to learn deep learning. I get polarized advice on it. Some argue that you need a degree or certificates from established MOOCs, others keep recommending me to do Kaggle challenges. Has anyone managed to land a decent deep learning job without formal CS/machine learning tra…

I felt like they were to math heavy. However, I'm struggling on how to learn deep learning. These statements are in contention. You will never really understand machine learning without learning a fair bit of the math. I do think a lot can be done on the presentation of the material, and certainly don't think much of credentialism. Honestly, in your shoes I would look for a position where you can learn from people in…

Thanks for your reply. I do agree with you, in general, and have been trying to get myself involved in more ML projects at my current work.

I have around 8 years of professional software experience (C++/C#) and have fiddled around with some rudimentary machine learning for work, like linear regression, k-means clustering, etc. I have a decent idea of how/why they work, but have fallen flat on my face when learning the theory behind more complicated algorithms, e.g. Hessians from Andrew Ng's class. In my experience, many classes tend to focus on a ground up approach. With higher level frameworks like Keras, how necessary is this?

Re: Coding the History of Deep Learning

#14

Post by yorwba »

dpcx wrote: post #6

This seems like a great introduction to the history. I have a problem with it, though. In the first example, the method compute_error_for_line_given_points is called with values 1, 2, [[3,6],[6,9],[12,18]]. Where did those values come from? Later in that same example, there is an "Error = 4^2 + (-1)^2 + 6^2". Where did those values come from? Later, there's another form: "Error = x^5 - 2x^3 -2" What about these? Ther…

The other replies are already telling you that these are just examples. I want to stress that these are completely unrelated examples, which is bad form IMO.

If the first example had been kept, then the second would have been "Error = (6 - (2·3 + 1))² + (9 - (2·6 + 1))² + (18 - (2·12 + 1))² = (-1)² + (-4)² + (-7)² = 66", which is what compute_error_for_line_given_points evaluates to.

The third would have been "Error = (6 - (m·3 + b))² + (9 - (m·6 + b))² + (18 - (m·12 + b))² = 3·b² + 42·b·m - 66·b + 189·m² - 576·m + 441" and its derivative would have to be taken in two directions, giving "dError/dm = 42·b + 378·m - 576" and "dError/db = 6·b + 42·m - 66". Visualizing that slope would require a 3D plot.

Re: Coding the History of Deep Learning

#15

Post by madenine »

emilwallner wrote: post #5
terrabytes wrote: post #2

Spot on. I struggled with the mainstream deep learning/machine learning MOOCs. I felt like they were to math heavy. However, I'm struggling on how to learn deep learning. I get polarized advice on it. Some argue that you need a degree or certificates from established MOOCs, others keep recommending me to do Kaggle challenges. Has anyone managed to land a decent deep learning job without formal CS/machine learning tra…

This is something I've also struggled with. I find it hard to read deep learning papers because I need to translate each math notation, thus struggling to get the bigger picture. I'm fond of the bottom-up approach, e.g. I started by mastering C and wrote my own libraries. But for deep learning I lean towards the opposite, starting with high-level libraries. When I want to understand the theory I search for simple pyt…

"Machine Learning Engineer" is a title we're going to see more and more of (and we're already seeing a lot).

Its one thing to know the math and theory to design, train, and tune the algorithm your company needs. But implementing it into production, at scale? That's not the same person.

Ideally, you have Person/Team A, who designs but knows enough about implementation to keep that in mind during their process, and Person/Team B who implements it into the software but knows enough about the design to make it work.

Re: Coding the History of Deep Learning

#16

Post by padthai »

sun_n_surf wrote: post #10

Least squares, gradient descent and linear regression separately? I get that he wants to point out the profundity and universality of the ideas encompassed in those techniques (& models; least squares and gradient descent are rightly thought of as (numerical) techniques, whereas the linear regression models is a, well, model) but that is like saying that arithmetic is fundamental to deep learning. Essentially, this "…

Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly?

y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β*

I might be missing something in the blog post.

Re: Coding the History of Deep Learning

#17

Post by luminati »

houqp wrote: post #8

Would love to see mention of several of the main contributors to deep learning, such as Geoffrey Hinton, the “father” of deep learning, Andrew Ng and Demis Hassabis in future posts.

Geoff Hinton - surely. But I think most experts will disagree on the other two. In terms of deep fundamental contributions I don't the think other two have made much. I think Andrew Ng has been a great popularizer/marketing guy - primarily with that Cats project. Likewise Demis Hassabis has been a great application creator - with amazing results of course - AlphaGo, Atari, etc.

On a side note: I lost all respect for andrew ng after the Baidu cheating scandal. https://www.nytimes.com/2015/06/04/technology/computer-scien...

I felt he got away too easy on that, without any apology or even a public statement - especially considering he is a former academic. (And that too he silently deleted his google+ posts.) Imagine if something like that had happened at a Google research team - I am pretty sure Jeff Dean or Peter Norvig would have stepped down.

Re: Coding the History of Deep Learning

#18

Post by narenst »

The article mentions that GPUs are on average 50-200 times faster for deep learning, I’m curious on how he came to that number. It has a lot to do with the code and the frameworks used. I haven’t come across a good comparison, most figures seems to be taken out of the blue.

Re: Coding the History of Deep Learning

#19

Post by tmoot »

padthai wrote: post #16
sun_n_surf wrote: post #10

Least squares, gradient descent and linear regression separately? I get that he wants to point out the profundity and universality of the ideas encompassed in those techniques (& models; least squares and gradient descent are rightly thought of as (numerical) techniques, whereas the linear regression models is a, well, model) but that is like saying that arithmetic is fundamental to deep learning. Essentially, this "…

Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly? y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β* I might be missing something in the blog post.

really quickly:

matrix inversion is ~O(n^3)

gradient descent is ~O(np) where p is the number of predictors and n are the observations (n x p matrix).

for lasso, calculating that derivative of the multiplier is not possible (for all points), so coordinated descent is used.

Re: Coding the History of Deep Learning

#20

Post by gajomi »

terrabytes wrote: post #12
ska wrote: post #3

Earlier quoted context omitted.

I felt like they were to math heavy. However, I'm struggling on how to learn deep learning. These statements are in contention. You will never really understand machine learning without learning a fair bit of the math. I do think a lot can be done on the presentation of the material, and certainly don't think much of credentialism. Honestly, in your shoes I would look for a position where you can learn from people in…

Thanks for your reply. I do agree with you, in general, and have been trying to get myself involved in more ML projects at my current work. I have around 8 years of professional software experience (C++/C#) and have fiddled around with some rudimentary machine learning for work, like linear regression, k-means clustering, etc. I have a decent idea of how/why they work, but have fallen flat on my face when learning th…

>With higher level frameworks like Keras, how necessary is this?

I would wager that you've heard this line before, but it all depends on the particulars of what you are trying to do. If you want to develop a first principles understanding of what's going on its probably important. It will be less important if you just need to see the empirical performance of n established method on your new dataset.

>but have fallen flat on my face when learning the theory behind more complicated algorithms, e.g. Hessians from Andrew Ng's class

Reading in between the lines, maybe this is a question about Newton's method? One of the general strategies shared between software development and "mathematical" (for lack of a better word) science and engineering is to reduce a complex problem to a known use case. If you've got a grasp on linear regression, take a look at Newton's method in this case. You may be pleasantly surprised to see that the Hessian is constant. This might make it easier to make the connection to relevant topics such as the convergence rate of the method and the connection to the uncertainty in the fit.

Post reply on HN