Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: The Hessian of tall-skinny networks is easy to invert

github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Show HN: The Hessian of tall-skinny networks is easy to invert

#21

Post by Lerc »

Nevermark wrote: post #15
Lerc wrote: post #13

I am not a mathematician, but I do enough weird stuff that I encounter things referring to Hessians, yet I don't really know what they are, because everyone who writes about them does so in terms that assumes the reader knows what they are. Any hints? The Battenburg graphics of matrices?

GRADIENT In the context of optimizing parameters of a model, the Gradient consists of all the derivatives of the output being optimized (i.e. the total error measure) with respect to each of the models parameters. This creates a simplified version of the model, linearized around its current parameter values, making it easy to see which direction to take a small step to move the ultimate output in the direction that i…

Thank you very much for this description.

If I understand it in a nutshell. If Gradient is the angle Hessian is the curvature.

and Jacobians let you know how much weights contributed to the blue component of something identified as a big blue cat.

I think.

Jacobians look like they could be used to train concept splitters. For instance if an LLM has a grab bag of possible conversation paths, the final embedding would have information for each path, but once the selection is made it could filter the embedding to that path, which would be beneficial for chain of thought using the filtered embedding instead of the predicted token. I always wondered how much the thinking in embedding space carried around remnants of conversation paths not taken.

Re: Show HN: The Hessian of tall-skinny networks is easy to invert

#22

Post by throwaway198846 »

hodgehog11 wrote: post #11
rahimiali wrote: post #5

Earlier quoted context omitted.

Good q. The method computes Hessian-inverse on a batch. When people say "Newton's method" they're often thinking H^{-1} g, where both the Hessian and the gradient g are on the full dataset. I thought saying "preconditioner" instead of "Newton's method" would make it clear this is solving H^{-1} g on a batch, not on the full dataset.

Just a heads up in case you didn't know, taking the Hessian over batches is indeed referred to as Stochastic Newton, and methods of this kind have been studied for quite some time. Inverting the Hessian is often done with CG, which tends to work pretty well. The only problem is that the Hessian is often not invertible so you need a regularizer (same as here I believe). Newton methods work at scale, but no-one with th…

I lately used these methods and BFGS worked better than CG for me.

Re: Show HN: The Hessian of tall-skinny networks is easy to invert

#23

Post by rahimiali »

petters wrote: post #16

Would be great to see this work continued with some training runs

Agreed. But these things have a way of not working out, and one the sadness, one forgets to celebrate the intermediate victories. I wanted to share an intermediate victory before reality crushes the joy.

Re: Show HN: The Hessian of tall-skinny networks is easy to invert

#24

Post by hodgehog11 »

throwaway198846 wrote: post #22
hodgehog11 wrote: post #11

Earlier quoted context omitted.

Just a heads up in case you didn't know, taking the Hessian over batches is indeed referred to as Stochastic Newton, and methods of this kind have been studied for quite some time. Inverting the Hessian is often done with CG, which tends to work pretty well. The only problem is that the Hessian is often not invertible so you need a regularizer (same as here I believe). Newton methods work at scale, but no-one with th…

I lately used these methods and BFGS worked better than CG for me.

Absolutely plausible (BFGS is awesome), but this is situation dependent (no free lunch and all that). In the context of training neural networks, it gets even more complicated when one takes implicit regularisation coming from the optimizer into account. It's often worthwhile to try a SGD-type optimizer, BFGS, and a Newton variant to see which type works best for a particular problem.
Post reply on HN