Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open source deep learning models that programmers can download and run first try

github.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Open source deep learning models that programmers can download and run first try

#11

Post by nahumfarchi »

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

That's interesting, I always thought that the graphical explanation was elaborate and confusing since a NN is just a bunch of matrices with non-linearities inbetween. To each his own I guess.

I definitely agree though that it's more of an experimental science at the moment.

Re: Open source deep learning models that programmers can download and run first try

#12

Post by _nx010_ »

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

They use matrices for computational efficiency. That's why linear algebra (along with diff equations and probability theory) is one of the prerequisites for any non-mooc machine learning course.

Re: Open source deep learning models that programmers can download and run first try

#14

Post by rs86 »

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

There is a clear theoretical reason for using 4 layers vs 3. It allows for more degrees of freedom which translates to a higher VC dimension. This implies numerous trade-offs in model behavior.

Besides this point, there is much more than simple derivatives in deep learning. For example regularization can yield quadratic programming problems. Different optimization algorithms can have tremendous impact on training time and model performance. Models can be quite sensitive to specific parameters that you can't just set at random.

More ingenious architectures like GAN also require some fairly technical thinking to get right. There is much more than image classification and vanilla NN or CNNs.

Re: Open source deep learning models that programmers can download and run first try

#16

Post by solomatov »

Retric wrote: post #15
solomatov wrote: post #13

It's a bad idea to learn deep learning without learning the math.

I disagree, now prove your point.

Your deep learning models don't always work as expected. In such cases you need to debug it. Understanding how models works internally is required for debugging them.

Re: Open source deep learning models that programmers can download and run first try

#17

Post by return0 »

Retric wrote: post #15
solomatov wrote: post #13

It's a bad idea to learn deep learning without learning the math.

I disagree, now prove your point.

Question: What is DL without the math? Network design? Stacking nodes in a graph in arbitrary ways? The days of handcracted design are numbered anyways, so learning DL without the math is not a good long term investment.

Re: Open source deep learning models that programmers can download and run first try

#18

Post by Houshalter »

rs86 wrote: post #14
reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

There is a clear theoretical reason for using 4 layers vs 3. It allows for more degrees of freedom which translates to a higher VC dimension. This implies numerous trade-offs in model behavior. Besides this point, there is much more than simple derivatives in deep learning. For example regularization can yield quadratic programming problems. Different optimization algorithms can have tremendous impact on training tim…

>There is a clear theoretical reason for using 4 layers vs 3. It allows for more degrees of freedom which translates to a higher VC dimension.

But then why does using 5 layers work worse than 4? Your theory is no good at predicting what the hyperparameters should be. The only way to find the correct hyperparameters is through empirical search.

>there is much more than simple derivatives in deep learning. For example regularization can yield quadratic programming problems. Different optimization algorithms can have tremendous impact on training time and model performance.

All these concepts are fairly simple also and can be expressed with little math. Additionally, a casual user doesn't need to have a deep understanding of them and the library will usually take care of it. Any more than a programmer needs to have a deep understanding of how an optimizing compiler works.

>More ingenious architectures like GAN also require some fairly technical thinking to get right.

The idea of using NNs to trick each other, is also fairly simple. It doesn't even involve any math.

Re: Open source deep learning models that programmers can download and run first try

#19

Post by return0 »

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

indeed the most unfamiliar/off-putting part might be the matrix formulation of the designs (and the matching of dimensions), which is not even useful if you are trying to implement a toy example in programming. But you can equally well understand backpropagation by following the updating of a single weight, which is much more intuitive.

The other thing is the unfortunate/misleading/atrocious jargon that has been adopted.

Re: Open source deep learning models that programmers can download and run first try

#20

Post by return0 »

davedx wrote: post #10
reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

Using matrices to perform the calculations is an optimization over doing a bunch of for loops. This vectorization results in faster code within higher level languages and on certain hardware platforms (SIMD). It's nothing to do with "slicker notation", although having written gradient descent with for loops and matrix operations, the vectorized version is simpler and cleaner to read in my opinion.

This has nothing to do with understanding backpropagation (which (correct me if i m wrong) is really the core of DL). In fact in the old days backpropagation was all about "propagating the deltas" and nothing about vectorizing.
Post reply on HN