NN failed to model sine function: https://towardsdatascience.com/can-machine-learn-the-concept...
Why train when you can optimize?
41–50 of 70 posts
Re: Why train when you can optimize?
#42I've always been interested in reinforcement learning, believing it will magically solve anything I throw at it. Unfortunately, I haven't got it to work everywhere yet. I hoped to learn a couple good RL algorithms and then never have to actually learn the concrete details of optimization, because RL can do almost as good. I don't truly believe this, but I think it is an underlying psychological reason for my love of…
I feel the exact same way!! Throw RL at the wall for any problem, and it will magically solve it. In the back of my mind, I know that it's not true, but I always want to throw RL at any problem. Right now, I am considering throwing it to make TCP/other-networking-stuff more efficient...
Re: Why train when you can optimize?
#43The article itself is good but the title is clickbait. The process of training is minimization (in other words, optimization) of generalization error.
Re: Why train when you can optimize?
#44The author claims ML is overused and many problems that could be solved more effectively with optimizations are solved using ML. If wonder what classes of problems fit this? For sure, I can't imagine how can you tackle sentiment analysis or text classifiers using optimization.
> tackle sentiment analysis
VADER (Valence Aware Dictionary and sEntiment Reasoner) is a large lookup table mapping words to a sentiment score, calculated by surveying people. It is simple to use and involves no ML.
Re: Why train when you can optimize?
#45A small shout-out to Differential Evolution, which is my go-to (derivative free) optimization algorithm. Think of a Genetic Algorithm, but the crossover operator is linear interpolation (DE's natural domain is real-valued vectors). It's simple, and in my experience, it works pretty well.
Scipy has a good implementation of that. I had some fun with it over the last couple of weeks. Just be aware that it runs a "polishing" optimization at the end which might improve the results, or just take forever :-)
Re: Why train when you can optimize?
#46The author claims ML is overused and many problems that could be solved more effectively with optimizations are solved using ML. If wonder what classes of problems fit this? For sure, I can't imagine how can you tackle sentiment analysis or text classifiers using optimization.
Supervised learning, which is one of the main type of ML is about learning from examples of input and desired output. In some cases it is easier to define the behavior of a function using such example pairs as opposed to specifying exactly what the learned function ought to do. This usually happens when we aren't certain what the right function value should be at every point in its input space . This is where ML should be used.
There are other functions, for example, searching, sorting etc where its easier to specify accurately what the desired function does compared to giving a list of pairs of examples. In such cases ML may not be the best choice. Note, reasonably accurate sorting functions can be learned from examples, but that's not the most efficient way to design a sorting function.
Re: Why train when you can optimize?
#47The article itself is good but the title is clickbait. The process of training is minimization (in other words, optimization) of generalization error.
True, but to be a bit pedantic, I wouldn’t call that generalization error (which you don’t have access to while training). Training minimizes some loss function which is a proxy for generalization error.
Re: Why train when you can optimize?
#48Isn't training literally an optimization of a loss function over training data?
It is and that is why I find the title a clickbait. The article isnt bad, but the title is certainly being coyly disingenuous.
In problems like this there are two aspects, (i) designing or specifying the search space of functions (ii) choosing the best function within the search space.
The opposite extremes are a) the search space contains only one function, the right function. In this case the training/optimization is moot. The other extreme is to have a very wide search space, say all smooth functions. In that case searching/training/optimizing is more challenging. The more reasonable example is one uses domain knowledge to design a much more restricted search space (for example, one may encode that the function is periodic with a known period) making the next step easier.
Re: Why train when you can optimize?
#49The article itself is good but the title is clickbait. The process of training is minimization (in other words, optimization) of generalization error.
True, but to be a bit pedantic, I wouldn’t call that generalization error (which you don’t have access to while training). Training minimizes some loss function which is a proxy for generalization error.
Spot on.
Re: Why train when you can optimize?
#50Earlier quoted context omitted.
I'm surprised to see almost no discussion of fourier series in that paper, considering fourier series is all about representing signals as linear combinations of sinusoidal functions.
You may be interested in [1] where they go to a great extend to show that the convolution operation that we consider in DL is the dual of fourier series [2]. [1] https://geometricdeeplearning.com [2] https://arxiv.org/pdf/2104.13478.pdf page 27 (23 if you count book pages).
Is convolution in DL not implemented with the FFT as the underlying workhorse?