Exploring Weight Agnostic Neural Networks

For instance, I've been working in users profiling and it's been a challenge to find which features and in which representation allow the model to learn. It's fantastic when you make a little change in a feature (for instance, use the median instead of the mean) and your model suddenly gets a +5% acc.

The field in the real world is not as simple as to make an API call. The data in the wild is really complicated. To get value from this data is the real challenge.

It's finding the right architectures that capture data priors and symmetries, and that can learn efficiently with stochastic gradient descent. Automatic differentiation is just a useful tool to use with those ideas.

Machine learning can be viewed through optimization, probablity and information theoretical lenses, and each of them will give you understanding of a different aspect. One recent "click" I had in my head came from debating with colleagues who are really strong in optimization: they routinely define values, sets, functions etc. as the end result of an optimisation problem. And this makes total sense.

Going down to maths that most people might be familiar with from highschool, think about how you define a line in space, or a plane. You write down "y=f(m)=a+mx" for a line (which you could read: to find a point m times the length of x away from the anchor point of a line, start at the anchor and move m times along vector x") "P={x:dot(w,x-x0)=0}" for a plane (read: the set of all x for which the dot product is 0). These are two different forms of describing the objects, one which is expressed as a function* using the constraints placed on the object to move along on it, and the other simply as a concise way to express all constraints. You can move from one form to another for both objects, and finding these types of connections increases your understanding of these objects and how their constraints define their structures.

Now, optimisation problems are similarly defined by their cosntraints,but they are much more flexible and connected to reality. We describe constraints in the hierarchy of convex (LP, QP, SOCP,SDP, conal) and nonconvex ( aka. "fucking difficult to deal with") constraints and then develop optimisers that attempt to fulfill these constraints as best as possible given some data. If it's solvable, then we have a way to reason explicitly about very complex properties the data and the mathematical objects contained in it - which can be used to solve actual problems, like finding the pareto frontier of a decision space, or laying out an UI nicely.

The machine learning takes this and pushes it to it's extreme, finding ways to navigate extremely high dimensional spaces and reason about the objects contained therein. Things like word vectors, latent representations and the "natural image manifold", finding ways to discover the "separation" of classes by building generative models and measuring distances in the latent space, visualisation techniques like curch plots, TSNE ...all of this extends our ability to reason about and understand data and ultimately our world.

That's one thing that excites me at least

But there was nothing new about architecture, essentially it is the choice of multidimensional function one tries to fit. In physics we have been fitting functions for hundreds of years. If you look at some experimental plot of say interference then you might decide to fit a sinusoid to it plus a background constant etc... the importance of fitting the right kind of function is obviously important, but we didn't know about algorithmic differentiation for hundreds of years (and it surely would have been welcome back then, even if performed by hand, it beats trial and error gradients).

That RM automatic differentiation is simple is easy to say in hindsight!

I don't think a richer diversity of functions is a bad idea, but it's already being used, softmax, exponents, sums, squares, ... why not perform gradient descent over a differentiable family of function that encompass these?

It's really disingenious to pretend RM AD was so very simple and then watch approvingly how someone throws it out the window and reverts to ... genetic programming? You want to let the computer find the best functions? fine, but then give the computer a superfunction which for certain values of an extra parameter differentiably reaches the functions you want to be considered.

Most of the architectures ... end up looking suspiciously much like plain old statistical physics! It's like we repeatedly witness how yet another introductory statistical physics expression turns out to perform well on very general sets of tasks (it really comes across as if everything should be treated like a dumb mole of water, and we never tried before because we simply refused to believe it could be that simple).