Is deep learning a new kind of programming?

Post by wiz21c » Sun, Dec 13, 2020, 8:13 AM UTC

m_mueller wrote: post #12

chriszhang wrote: post #7

We trained a deep learning model to look at like 20 system parameters and predict an output. the parameters were binary. So one curios engineer decided to brute-force the trained model with all possible inputs like 2^20 inputs to see what the model does. he found for the problem we were solving only 4 of the 20 parameters had effect on results. the remaining approx 16 parameters do not affect results. So he replaced…

For up to 100ish parameters, even mixed with floating point I recommend trying the midaco solver a friend of mine develops. MINLP, ant colony method (i.e. gradient descent with many restarts). From my experience this runs circles around NNs for this class of problems (parameter optimization with relatively low complexity and/or limited amount of training data available).

Post by freeone3000 » Sun, Dec 13, 2020, 8:26 AM UTC

xapata wrote: post #6

hprotagonist wrote: post #4

I have been writing optimization solvers of many forms to solve problems in engineering for about 20 years. from "make excel do linear regression on some data" to linear least squares to some nonlinear methods, simulated annealing, bayesian methods, deep neural networks -- none of this is "a new kind of programming", it's "do a bunch of data munging, throw matrix at a function, get matrix back, interpret/plot." there…

Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different. Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.

Post by f6v » Sun, Dec 13, 2020, 8:53 AM UTC

ipsum2 wrote: post #5

hprotagonist wrote: post #4

I have been writing optimization solvers of many forms to solve problems in engineering for about 20 years. from "make excel do linear regression on some data" to linear least squares to some nonlinear methods, simulated annealing, bayesian methods, deep neural networks -- none of this is "a new kind of programming", it's "do a bunch of data munging, throw matrix at a function, get matrix back, interpret/plot." there…

How well does "linear least squares to some nonlinear methods, simulated annealing, bayesian methods" work when you're doing speech to text, text generation or object detection? Deep learning is different in that it's a huge leap closer to human capabilities compared to the other methods you've listed.

Post by xapata » Sun, Dec 13, 2020, 9:01 AM UTC

freeone3000 wrote: post #23

xapata wrote: post #6

Earlier quoted context omitted.

Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different. Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.

The feedback machine is also represented as a matrix, with either implicit or explicit recurrence by the code around the model.

Post by xapata » Sun, Dec 13, 2020, 9:09 AM UTC

_nhynes wrote: post #18

xapata wrote: post #6

Earlier quoted context omitted.

Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different. Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.

RNNs provided by common frameworks really do give a matrix back (examples [0, 1]). They're about as stateful as any other object or generator function. [0]: https://pytorch.org/docs/stable/generated/torch.nn.RNN.html [1]: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cu...