Computing Higher Order Derivatives of Matrix and Tensor Expressions [pdf]

Post by SoerenL » Sun, Nov 18, 2018, 7:54 PM UTC

jkam wrote: post #18

SoerenL wrote: post #14

Earlier quoted context omitted.

Not in its current formulation. It uses a different representation of the tensors. However, a new version/algorithm that will be available in a few months can be used in TF and PyTorch.

Are you referring to the Ricci notation when you are saying it uses a different representation of the tensors? Do you also plan to add non-differentiable functions like relu?

Yeah, I am referring to Ricci notation. TF and PyTorch don't use it. The new version already has relus. This version is targeted at standard formulas (has abs as a non-differentiable function), next version works for deep learning. Never wanted to work on this but there is just no way around it.

Post by bmc7505 » Sun, Nov 25, 2018, 3:44 AM UTC

jkam wrote: post #19

bmc7505 wrote: post #17

Would be interesting to explore whether evaluating ∇²f(x) for higher-order SGD methods directly is now feasible for smaller DNNs and whether this leads to faster convergence during training. Most methods like Newton or Gauss-newton were thought intractable for DNNs. Also curious if the angle descent prescribed by quasi-Newton methods is empirically closer to the true 2nd order gradient and whether these approximation…

What kind of work are you referring to when you say higher-order SGD may _now_ be feasible for deep learning? I only find results that try to approximate second order information.

Not sure what you mean. The paper above claims 1000x speedups for computing second-order derivatives. Have not tested their claims, but was speculating that such an improvement, if true, would make computing hessians for small networks fesiable. This is what I am referring to.

Re: Computing Higher Order Derivatives of Matrix and Tensor Expressions [pdf]

Re: Computing Higher Order Derivatives of Matrix and Tensor Expressions [pdf]