Why train when you can optimize?

I like your article and went to your home page to find more good articles and I like what I saw. Thank you for sharing. The only thing is that I am reading from my phone and the site is not very mobile friendly

> It’s very easy to train a neural network to look at a piece of a sine wave and predict what the next value should be - the fact that the function is periodic actually helps you.

I’m not sure what you mean. Are you talking about training a network to predict cos(x)? In this case nothing at all changes by training on the derivative.

Or do you mean train the net to take as input sin(t) and produce sin(t+0.01)? The problem with this is that any given value of sin(t) has 2 answers for sin(t+0.01), therefore your optimizer is going to spit out 0 as the answer. Plus, this is a different problem entirely, and you lose the ability to infer sin(t) based on t. It doesn’t answer the same question.

Your suggestion is further going to be seriously confounded if the periodic function is more complex, say, the sum of several sin waves. There can be an arbitrary number of values y that correctly match f(x).

You should try it before making assumptions that it’s easy, see what it takes to train a net to predict sin(x), without embedding knowledge of the fact that sin is periodic.

> The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of state.

This also sounds like an assumption to me that somehow the entire world of research has failed to consider the most obvious of ideas. The point of both optimizers and neural networks is that they can be black boxes, right? It doesn’t matter at all whether the input is time based or position based or a function of money. The network, in theory, can learn any function, time or otherwise, and there’s nothing special about time.

But, neural networks function better with domain knowledge. When you know the function domain is time and that the output is periodic, you can do things to make a network easier to train, like using a periodic activation function.

As a side note, RNNs explicitly model an NN based on previous state. Also all layered NNs can be viewed as a series of smaller nets that feed state to the next net. In some sense, NNs always evaluate as a function of state.

wavenet and transformer models already come dangerously close

> I might take up your challenge and write a blog post about it or something if I get any success.

Success or failure, I would really enjoy seeing that write up! I would be even more excited to be proven wrong.

The promise of "universal function approximator" is very temping. My personal dream would be to have it so one could essentially run scipy in reverse and learn the entire library with a NN. Even in the case I gave, the idea that you could learn an arbitrary quantile function means you could also arbitrarily learn a sampler for any distribution, since all you have to do compose a uniform sampler with whatever quantile function you learned.

Of course for this example I'm using "solved" using a similar approach with variational inference (pyro has a write up on it: https://pyro.ai/examples/svi_part_i.html, you might find David Blei's "Variational Inference: A Review for Statisticians" useful as well https://arxiv.org/abs/1601.00670)

I don't consider VADER as suitable for tackling sentiment analysis but perhaps we have a different understanding of 'tackling sentiment analysis'. In my view, 'tackling sentiment analysis' requires doing sentiment analysis well, which no systems currently do.

Decent sentiment analysis is a yet unsolved problem and current state of art solutions (mostly based on large pretrained langugage models like BERT, leveraging transfer learning from very large unlabeled corpora) are not sufficiently accurate, and of course neither is VADER. As soon as you go beyond simple binary polarity (blatantly positive/negative sentiment) and something more informative like various five-way sentiment tasks, or aspect-based sentiment or determining sentiment targets, sentiment analysis still has a long, long way to go.

Current ML solutions are a possible way to tackle sentiment analysis because while the current models are not good enough, we have evidence showing that with increased model and data size they improve and perhaps might result in sufficiently good sentiment analysis someday. Perhaps not, and we'll need something else; but at least they have some potential.

VADER and similar methods are not a way to tackle sentiment analysis because not only the current VADER model is not nearly good enough, it's clear that they can't scale to anything much better than that. Adding extra surveys to increase the lookup table quickly hits diminishing returns (unlike ML systems where we consistently see that ever larger models continue to get improvements), and if they can't even reach the current bar of ML models (which still are not good enough) then they can't bring us to the level of sentiment analysis where we want to be, they are a dead end.

Tackling sentiment analysis also requires handling a wide variety of languages, not only English, which is yet another aspect where data-driven models have an advantage over systems that require extensive human labor for each new language.

I mean, we all (me included!) want to believe that human knowledge encoded in rules can work, intuitively it's a very appealing concept, however, it does not work out in practice and IMHO in the end we all have to learn to accept the Bitter Lesson (http://incompleteideas.net/IncIdeas/BitterLesson.html).