Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding the History of Deep Learning

blog.floydhub.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Coding the History of Deep Learning

#21

Post by gipp »

padthai wrote: post #16
sun_n_surf wrote: post #10

Least squares, gradient descent and linear regression separately? I get that he wants to point out the profundity and universality of the ideas encompassed in those techniques (& models; least squares and gradient descent are rightly thought of as (numerical) techniques, whereas the linear regression models is a, well, model) but that is like saying that arithmetic is fundamental to deep learning. Essentially, this "…

Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly? y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β* I might be missing something in the blog post.

You can, yes, but inverting a matrix is computationally expensive, and for a large X, numerical optimization methods can be much more time/space efficient.

Re: Coding the History of Deep Learning

#22

Post by snakeboy »

padthai wrote: post #16
sun_n_surf wrote: post #10

Least squares, gradient descent and linear regression separately? I get that he wants to point out the profundity and universality of the ideas encompassed in those techniques (& models; least squares and gradient descent are rightly thought of as (numerical) techniques, whereas the linear regression models is a, well, model) but that is like saying that arithmetic is fundamental to deep learning. Essentially, this "…

Also why do linear regression (OLS) models need gradient descent at all? Cannot you calculate the parameters directly? y = X β + ε ...and a few assumptions give you... (X^t X)^-1 y = β* I might be missing something in the blog post.

Others have pointed out that matrix inversion is O(n^3) and hence computationally infeasible. It is also worth considering that the conditioning of X^t X, k(X^t X) can be as large as k(X)^2, so solving in this way can be very unstable.

Re: Coding the History of Deep Learning

#23

Post by netvarun »

iluvmylife wrote: post #4

If you want a more nuanced research on the history on deep learning in neural networks, here is an excellent historical survey paper: https://arxiv.org/abs/1404.7828

I'd also recommend Andrey Kurenkov's well written multi-part series on the history of neural nets: https://news.ycombinator.com/item?id=10910887

[The author of the paper mentioned on parent comment is Jürgen Schmidhuber - inventor of LSTMs and a very colorful character in neural land. The NYTimes did a nice profile on him a while back: https://www.nytimes.com/2016/11/27/technology/artificial-int... HN Discussion: https://news.ycombinator.com/item?id=13066646]

Re: Coding the History of Deep Learning

#24

Post by twillmas »

dpcx wrote: post #6

This seems like a great introduction to the history. I have a problem with it, though. In the first example, the method compute_error_for_line_given_points is called with values 1, 2, [[3,6],[6,9],[12,18]]. Where did those values come from? Later in that same example, there is an "Error = 4^2 + (-1)^2 + 6^2". Where did those values come from? Later, there's another form: "Error = x^5 - 2x^3 -2" What about these? Ther…

I'd also like to see more of a "teaching" post that can walk through the math incrementally.

Many of the deep learning courses assume "high school math", but my school must have skipped matrices, so I've been watching Khan Academy videos.

Are there any good posts / books on walking through the math of deep learning from a true beginner's perspective?

Re: Coding the History of Deep Learning

#26

Post by houqp »

luminati wrote: post #17
houqp wrote: post #8

Would love to see mention of several of the main contributors to deep learning, such as Geoffrey Hinton, the “father” of deep learning, Andrew Ng and Demis Hassabis in future posts.

Geoff Hinton - surely. But I think most experts will disagree on the other two. In terms of deep fundamental contributions I don't the think other two have made much. I think Andrew Ng has been a great popularizer/marketing guy - primarily with that Cats project. Likewise Demis Hassabis has been a great application creator - with amazing results of course - AlphaGo, Atari, etc. On a side note: I lost all respect for…

From a pure technical point of view, I agree with you. But there is no doubt they all have played an important role in popularising deeplearning. I am fancisnated in the history of deeplearning and how it went from a field no one cared to what it is today.

Re: Coding the History of Deep Learning

#27

Post by kahnjw »

luminati wrote: post #17
houqp wrote: post #8

Would love to see mention of several of the main contributors to deep learning, such as Geoffrey Hinton, the “father” of deep learning, Andrew Ng and Demis Hassabis in future posts.

Geoff Hinton - surely. But I think most experts will disagree on the other two. In terms of deep fundamental contributions I don't the think other two have made much. I think Andrew Ng has been a great popularizer/marketing guy - primarily with that Cats project. Likewise Demis Hassabis has been a great application creator - with amazing results of course - AlphaGo, Atari, etc. On a side note: I lost all respect for…

I didn't even realize this happened. Thank you for posting. Sucks because now I have less respect for Andrew.

I don't understand why you'd want to cheat for a competition like this? I get it, people cheat all the time, but the field of machine learning is built on a foundation of open and shared research, and trust.

Re: Coding the History of Deep Learning

#29

Post by AndrewOMartin »

kahnjw wrote: post #27
luminati wrote: post #17

Earlier quoted context omitted.

Geoff Hinton - surely. But I think most experts will disagree on the other two. In terms of deep fundamental contributions I don't the think other two have made much. I think Andrew Ng has been a great popularizer/marketing guy - primarily with that Cats project. Likewise Demis Hassabis has been a great application creator - with amazing results of course - AlphaGo, Atari, etc. On a side note: I lost all respect for…

I didn't even realize this happened. Thank you for posting. Sucks because now I have less respect for Andrew. I don't understand why you'd want to cheat for a competition like this? I get it, people cheat all the time, but the field of machine learning is built on a foundation of open and shared research, and trust.

On skim-reading the article it seems they were banned from submitting entries to a competition server for 12 months because they made a significant number of submissions.

It's arguable that they were gaming the system somewhat, but unless a limit was explicitly defined then this just seems like they were doing a lot of exploration in the area.

Imagine if you published some research showing you'd made something that did something cool, but then people lost respect for you because you'd made a lot of previous attempts.

Re: Coding the History of Deep Learning

#30

Post by interknot »

AndrewOMartin wrote: post #29
kahnjw wrote: post #27

Earlier quoted context omitted.

I didn't even realize this happened. Thank you for posting. Sucks because now I have less respect for Andrew. I don't understand why you'd want to cheat for a competition like this? I get it, people cheat all the time, but the field of machine learning is built on a foundation of open and shared research, and trust.

On skim-reading the article it seems they were banned from submitting entries to a competition server for 12 months because they made a significant number of submissions. It's arguable that they were gaming the system somewhat, but unless a limit was explicitly defined then this just seems like they were doing a lot of exploration in the area. Imagine if you published some research showing you'd made something that d…

The specified limit was 2 submissions per week, according to the people organizing the competition:

http://www.image-net.org/challenges/LSVRC/announcement-June-...

Post reply on HN