Open source deep learning models that programmers can download and run first try • Hacker News Board

Re: Open source deep learning models that programmers can download and run first try

#21

Post by Houshalter » Fri, Jun 16, 2017, 5:06 PM UTC

davedx wrote: post #10

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

Using matrices to perform the calculations is an optimization over doing a bunch of for loops. This vectorization results in faster code within higher level languages and on certain hardware platforms (SIMD). It's nothing to do with "slicker notation", although having written gradient descent with for loops and matrix operations, the vectorized version is simpler and cleaner to read in my opinion.

He's not complaining about using vectorization in code. The problem is papers and even explanations targeted at non-experts, often use obfuscated math in place of clear explanations. I've complained about this before here: https://news.ycombinator.com/item?id=13953530

Mathematical notation is basically a programming language. A programming language with weird symbols you can't type to search for, single letter variable names for everything, and no comments. And it's written by programmers that are obsessed with fitting everything into a simple line and making it as small as possible, no matter how difficult it is to read. Any programmer understands this is incredibly bad practice. And even if parse every step and perfectly follow what the code is doing, without explanation, it's pretty difficult to figure out why.

Re: Open source deep learning models that programmers can download and run first try

#22

Post by Houshalter » Fri, Jun 16, 2017, 5:15 PM UTC

_nx010_ wrote: post #12

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

They use matrices for computational efficiency. That's why linear algebra (along with diff equations and probability theory) is one of the prerequisites for any non-mooc machine learning course.

I've programmed neural networks without knowing any linear algebra. When I needed to figure out how to use vector operations for speed, it took like 5 minutes to search for matrix multiplication on wikipedia. You can't get by without even knowing that, as element wise operations can do everything just as fast.

Re: Open source deep learning models that programmers can download and run first try

#23

Post by return0 » Fri, Jun 16, 2017, 5:16 PM UTC

nahumfarchi wrote: post #11

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

That's interesting, I always thought that the graphical explanation was elaborate and confusing since a NN is just a bunch of matrices with non-linearities inbetween. To each his own I guess. I definitely agree though that it's more of an experimental science at the moment.

OTOH thinking in matrices only may be limiting, as there are potential designs one might want to try (brain-inspired for example) that cant be expressed in matrix operations.

Re: Open source deep learning models that programmers can download and run first try

#24

Post by jcl » Fri, Jun 16, 2017, 5:38 PM UTC

rrggrr wrote: post #8

Gosh I wish this existed for Python.

Could you clarify? Several of the examples are Python projects (with and without Tensorflow), and others are apps consuming models that probably came from Tensorflow.

Re: Open source deep learning models that programmers can download and run first try

#25

Post by [deleted] » Fri, Jun 16, 2017, 5:51 PM UTC

solomatov wrote: post #16

Retric wrote: post #15

Earlier quoted context omitted.

I disagree, now prove your point.

Your deep learning models don't always work as expected. In such cases you need to debug it. Understanding how models works internally is required for debugging them.

[deleted]

Re: Open source deep learning models that programmers can download and run first try

#26

Post by joshvm » Fri, Jun 16, 2017, 6:06 PM UTC

solomatov wrote: post #16

Retric wrote: post #15

Earlier quoted context omitted.

I disagree, now prove your point.

Your deep learning models don't always work as expected. In such cases you need to debug it. Understanding how models works internally is required for debugging them.

I would wager that a lot of people who "do deep learning" have absolutely no idea about the models they're using.

Hyperparameter optimisation is basically a fudge right now - you try everything and see what works. Even the research groups who came up with the standard network stacks, like VGG, basically lucked out and found an architecture that worked, then tried several variants and found one that worked better. DL papers are full of handwaving speculation about why particular networks perform better than others, but right now it's just that: highly educated speculation.

This isn't limited to deep learning. If you want to try any kind of machine learning, it's totally reasonable to throw different fitting functions at your problem to see which one works best. Unless you have an unusually clear problem category, it's rarely possible to say at the outset that "This problem would best be solved with method ". A counter here would be that if you need to classify images, you should almost certainly use a convnet.

You need some understanding about why things might be going wrong, e.g. your loss isn't moving -> crank up the learning rate. You're seeing nans? Probably your learning rate is too high. But that doesn't really need any serious maths to understand. You can get by quite well by figuring out empirical rules.

I'm not arguing that you shouldn't learn the maths, it's a wise idea to, but many people use deep learning models without knowing how backpropagation works for instance.

Re: Open source deep learning models that programmers can download and run first try

#27

Post by gnaritas » Fri, Jun 16, 2017, 6:08 PM UTC

Houshalter wrote: post #21

davedx wrote: post #10

Earlier quoted context omitted.

Using matrices to perform the calculations is an optimization over doing a bunch of for loops. This vectorization results in faster code within higher level languages and on certain hardware platforms (SIMD). It's nothing to do with "slicker notation", although having written gradient descent with for loops and matrix operations, the vectorized version is simpler and cleaner to read in my opinion.

He's not complaining about using vectorization in code. The problem is papers and even explanations targeted at non-experts, often use obfuscated math in place of clear explanations. I've complained about this before here: https://news.ycombinator.com/item?id=13953530 Mathematical notation is basically a programming language. A programming language with weird symbols you can't type to search for, single letter variab…

> Mathematical notation is basically a programming language.

A very bad one that can only be executed by brains with the requisite existing historical knowledge; in fact it's more like bad pseudo-code that lacks the explicitness necessary to translate into actual instructions. It's basically condensed jargon intended for the already converted.

It'd probably be vastly easier to teach math with an actual programming language than with traditional notation. Scheme would be ideal for this.

Re: Open source deep learning models that programmers can download and run first try

#28

Post by whytaka » Fri, Jun 16, 2017, 7:10 PM UTC

solomatov wrote: post #13

It's a bad idea to learn deep learning without learning the math.

I understand the chain-rule. I understand using the derivative to minimize error. What I want to know is, what math is applied where to architect better models? Preferably with examples.

Re: Open source deep learning models that programmers can download and run first try

#29

Post by cr0sh » Fri, Jun 16, 2017, 7:17 PM UTC

davedx wrote: post #10

reader5000 wrote: post #6

There isnt really any math to deep learning other than the concept of a derivative which is taught in high school calculus. The reason deep learning papers seem mathy is people take network architectures and various elementary operations on them and try to express them symbolically in latex using summations and indexing-hell. For example the easy concept of "updating all the neurons in one layer based on the neurons…

Using matrices to perform the calculations is an optimization over doing a bunch of for loops. This vectorization results in faster code within higher level languages and on certain hardware platforms (SIMD). It's nothing to do with "slicker notation", although having written gradient descent with for loops and matrix operations, the vectorized version is simpler and cleaner to read in my opinion.

I think OP means vector notation in the research papers, not using vector operations and representations at the code level.

That said, it can be useful for the beginner to implement a basic NN library from the ground up, to understand how the vector processing works, as well as what is going on step-wise with backprop and such.

Once that is fully understood, the next step of utilizing true vector processing libraries can be taken, and so on - eventually culminating in using and understanding libraries like TensorFlow.

Having the background of the lower levels gives you an appreciation and even insights when you transition to higher level frameworks.

That's just my opinion, though.

Re: Open source deep learning models that programmers can download and run first try

#30

Post by sctb » Fri, Jun 16, 2017, 7:21 PM UTC

itg wrote: post #4

shoshin23 wrote: post #3

I would say that this title is misleading. A lot of what is presented there needs a strong grasp of deep learning(and the other underlying concepts behind them.), without which all you'll do is load the examples on Xcode and run them. Moreover, I would probably encourage people to read examples of Tensorflow or Caffe2 running on iOS rather than something like Forge. Forge is an interesting project but won't really he…

The original title, which is also the title of the repo, was much more accurate. The author mentions these are models to download and start playing with right away, not a set of repositories to help you learn deep learning.

OK, we've updated the title from the (slightly edited) repo description of “Examples to get started with Deep Learning without learning any of the math” to this phrase from the description.