Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning works spectacularly well, but mathematicians aren’t sure why

quantamagazine.org

Post reply on HN
Print view View original

101–110 of 137 posts

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#101

Post by chestervonwinch »

the_duck wrote: post #77

>In the last 15 years or so, researchers have created a number of tools to probe the geometry of these hidden structures. For example, you might build a model of the surface by first zooming in at many different points. At each point, you would place a drop of virtual ink on the surface and watch how it spread out. It sounds to me like this 'ink drop' is a metaphor to explain some state-of-the-art dimensionality redu…

There are diffusion based methods for unsupervised learning based on the graph laplacian, or heat kernel [0]. Maybe this is what she was referring to? I would guess so, because Coifman (who is also closely tied to wavelets [1]) introduced the diffusion maps [2] method. On the other hand, I have not seen any unsupervised method based on say, Navier-Stokes, which is the only other thing that comes to mind for regarding ink drops diffusing.

[0]: http://www.mit.edu/~9.520/spring11/slides/class20_misha.pdf

[1]: https://en.wikipedia.org/wiki/Coiflet

[2]: https://en.wikipedia.org/wiki/Diffusion_map

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#102

Post by shas3 »

joakleaf wrote: post #15

Neural networks is advanced curve fitting -- That's why. It isn't really all that magical or mysterious.

That isn't saying much. Every supervised learning algorithm can be cast as a problem of advanced curve fitting, because at the end of the day, your 'curve' is the mapping that takes you from your 'x-axis' of data to the 'y-axis' of labels.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#103

Post by goblin_got_game »

modeless wrote: post #80
TTPrograms wrote: post #73

Earlier quoted context omitted.

The why is largely why does gradient descent converge to a good answer instead of getting stuck in a local minima.

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this…

This intuition is not correct. For a random point in a high-dimensional space--yes--I totally agree it's highly unlikely all the eigenvalues will have the same sign (as is needed for a local minimum). The problem with Surya's claim is that the set needs to be restricted to just the critical points (where the derivative is zero). It's very hard to describe the spectral properties generally for this set, and there has been results for only 2 and 3 dimensions as far as I'm aware. Maybe the measure is the same, at least for Deep Nets, and thus a critical point is as unlikely to be a local minima as a random point is. But no work has shown that. Furthermore, we know the number of critical points to be exponential for broad classes of functions so the subset cannot be ignored.

ELI5 version: One can ask, what's the probability that a random person drawn from Earth's population (points in high dimensions) owns a Bugatti (is a local minimum)? It's very small, obviously. But that doesn't tell us anything about the probability of Bugatti ownership among select subsets of people (critical points).

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#104

Post by crazypyro »

argonaut wrote: post #74
crazypyro wrote: post #20

Earlier quoted context omitted.

Yes, people took the Universal approximation theorem[0] as evidence that they only need 1 layer, but there is zero guarantee of efficiency. A single hidden layer may mean many magnitudes more neurons needed over a n-hidden layer, n > 1 network, which could cause an unrealistic training time. Having multiple layers can reduce this training time with an optimal structure. [0] https://en.wikipedia.org/wiki/Universal_app…

Nobody ever took that theorem seriously. Deep nets were around since the 90s.

I've seen people with passing knowledge of NNs throw it around sometimes and its also referred to in a lot of literature as one reason for needing to parallelize neural networks (which I've been reading a lot on, due to a project), even if its not hugely important.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#105

Post by rdlecler1 »

algomanic wrote: post #26
rdlecler1 wrote: post #23

One problem with understanding ANNs is that the weight matrix carries a lot of spurious interactions. Running perturbation analysis you can see that many of the interactions do not contribute to the information processing of the circuit. This is the same for Gene Regulatory Networks. I wrote a paper published in Nature's "Systems Biology" entitled Survival of the Sparsest Gene Networks are Parsimonious. It's been cit…

Could you explain weight matrix "spurious interactions"?

In short, a spurious interaction is a non-zero entry in the weight matrix that has no (positive) contribution to the network function, and that removing that interaction the network will perform at least as good as it would with the interaction.

Most neural network models assume that all neurons in one level are fully connected to all neurons in the next level. This leads to confusion about how ANNs work.

From my research I'd argue that MOST interactions in these networks are spurious. Once you remove them it reveals the (visual) topology (circuit diagram) that's driving the function of that network.

In the paper I wrote, I evolved gene regulatory networks (of ANNs have the same mathematical representation) such that interactions between any two nodes could be deleted, created (if W_ij = 0), or modified according to probabilities of deletion, creation, and modification. Given these probabilities, you can calculate the number of interactions that should result when the network reaches equilibrium, however what I found was that the network evolves less interactions than you would expect from the equilibrium calculation. This says that all things being equal, a network is paying a price for spurious interactions and that these will be removed in an evolutionary environment. Basically, each interaction needs to pay it's way otherwise it leads to unnecessary complexity that reduced the fitness of the network.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#106

Post by alexmlamb »

tel wrote: post #21
senekerim wrote: post #19

Earlier quoted context omitted.

It's not really a cheat, as it produces the MLE (i.e. most likely) estimate under certain assumptions (e.g. errors are normally distributed, which occurs naturally if the errors are large sums of many unrelated measurements errors).

Oh, I'm aware. It's just hard to always justify it and especially hard to do so in historical context.

I don't think that this is quite the right explanation.

|y - wx| isn't differentiable but it is subdifferentiable. So the derivative is defined at every point except where y = wx, and in that case you're more or less fine if you just pretend that the derivative is zero at that point.

One reason why squaring is preferred is that the derivative has a nice closed form, which can be used to find closed form solutions.

Another is that people like to fit their models with gradient descent, and gradient descent has better guarantees for strongly convex loss functions. It also works better in practice. Intuitively: if you try to minimize x^2 by gradient descent, you have the largest gradients when you're far from the minimum. If you try to minimize |x|, then your gradient is always either +1 or -1, making it harder to converge around the minimum. See the Huber loss for a strongly convex relaxation of the absolute value function.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#107

Post by j2kun »

dave_sullivan wrote: post #56

In addition to "We need a cluster for deep learning", the second most popular mostly untrue thing I hear is "We have no idea how neural networks learn". However, there are many papers that explore various ways to make a network learn, and they keep improving on performance, suggesting they're on to something . There are also many papers that discuss possible theoretical implications of experimental results. But what…

> What would a satisfactory "why" even look like exactly? As in, what form might it take compared to some other scientific discipline where we do know what's going on?

To me this is obvious: a proof is an adequate answer. All of the explanations given in this area are heuristic reasons. They're nice, but we can't know if they're correct or if we're being fooled by our intuition without a proof.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#108

Post by MrQuincle »

stdbrouw wrote: post #3
sonabinu wrote: post #2

It would be interesting to see what the purists arrive at

Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are "maximum likelihood estimates." Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC). Of course, many ma…

1.) Dropout.

Gal and Ghahramani "We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to the probabilistic deep Gaussian process" http://arxiv.org/pdf/1506.02142.pdf

2.) Deep.

There have been many hierarchical networks in Bayesian statistics. Hierarchical Dirichlet Process, hierarchical beta process, Pitman-Yor process, Gaussian process, nested, dependent, translated, generalized versions of the Chinese Restaurant process, or the Indian Buffet one. The deep part isn't new.

3.) Auto-encoders.

People seem to talk about auto-encoders a lot. Its very definition introduces hidden variables about which very little is defined. It corresponds with some kind of prior on what representation should be used by the network. This is studied in Bayesian compressive sensing. http://machinelearning.wustl.edu/mlpapers/paper_files/icml20...

4.) Layered training.

Layers are trained one by one. Doesn't really ring a bell. Maybe it corresponds to some MCMC method I'm not aware of. There are often inner and outer loops in MCMC and I guess something fundamental can be told about the right moments to switch from inner to outer. However, that does not correspond to learning low-level features first.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#109

Post by sdenton4 »

bdhe wrote: post #66
wsxcde wrote: post #50

To take the discussion on a slight tangent, how uncommon is this phenomenon? An applied tool works really well but nobody knows why. I can give another example from the domain of formal verification: SAT solvers which are at the core of most modern verification/synthesis tools. You can download open source SAT solvers today that work spectacularly well on "real" SAT instances with millions and millions of variables a…

> So this must there is some characteristic of the problems that we generate in practice that makes them "easy" but we don't have a good understanding of what this characteristic is. All we know, for now, is that we've somehow stumbled upon a near-perfect set of heuristics that work amazingly well on the SAT instances we encounter in practice. There's also the intriguing possibility that "natural instances" however o…

The 'other' possibility is that hard instances have 'measure zero' in the space of all instances... So when you pick some random problem in the real world, it ends up being relatively easy to solve with high probability.

My feeling is there's a bit of each; the problem space has a bit of unknowable quasi-structure, and also tends to miss the perverse corner cases and counterexamples that we like to think about as mathematicians.

Re: Machine learning works spectacularly well, but mathematicians aren’t sure why

#110

Post by deepnet »

HockeyPlayer wrote: post #79
vonklaus wrote: post #52

There was a great article called The Space Doctor's Big Idea , published in the New Yorker a few months back. It explained Einstein's theories in the top 1000s English words used in America. While sone might wave their hands and call this an eli5, it was quite well done and between that and some xkcd comics I was able to learn enough to ubderstand how orbit works and a few other awesome facts. I am interested in find…

You may enjoy The Master Algorithm: http://www.amazon.com/Master-Algorithm-Ultimate-Learning-Mac...

Google Talk on the Master Algorithm by the author Domingas.

https://www.youtube.com/watch?v=B8J4uefCQMc

Post reply on HN