Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beyond automatic differentiation

ai.googleblog.com

Post reply on HN
Print view View original

31–37 of 37 posts

Re: Beyond automatic differentiation

#31

Post by ftxbro »

ogogmad wrote: post #26
ftxbro wrote: post #22

Earlier quoted context omitted.

I see what you mean, but say you have such a matrix and there are two eigenvalues that you can't separate from each other. Then wouldn't their corresponding eigenvectors define like a 2d plane? Couldn't there still be some interval for that plane, like say that the plane is defined by the linear combination of two vectors and both vectors have zero on some coordinate, say the x coordinate. The interval could still be…

That makes sense, but what can be done with it?

sorry I'm not sure what you mean

Re: Beyond automatic differentiation

#32

Post by modeless »

6gvONxR4sf7o wrote: post #27
modeless wrote: post #11

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity…

So what? This isn't some kind of gotcha. They're being explicit about the limitation. The built an interesting thing and showed that it does interesting things in a toy setting. This gets me excited about future work down this path.

I disagree that the blog post was explicit enough. Tacking parenthesized disclaimers onto the end of your sentences is not really very explicit. What they're comparing against isn't even Adam; calling it Adam at all is misleading. The subtitle of the Adam paper is "A Method for Stochastic Optimization". It's not designed or intended for full-batch, non-stochastic training.

What they should do is lead that paragraph with the fact that this is full-batch, note that this is not normal, and explain why they chose to do it anyway. And they shouldn't call it plain "Adam".

The paper is probably good, and it's fine to test things in toy settings, but you need to be upfront with your assumptions and limitations.

Re: Beyond automatic differentiation

#35

Post by [deleted] »

nyrikki wrote: post #24
bollu wrote: post #21

Earlier quoted context omitted.

Could you provide a link to some more reading about this? i do not know what a wada property is, nor why it is topological

The Wada property is when you have 3+ disjoint sets that share the same boundary set, at least for the complete Wada property. In Newton's fractal, any circle you draw will either have a single root or all roots, no matter how small you draw that circle. If your initial conditions are near that boundary the exit basin you take is indeterminate because that point wise boundary point is the boundary of multiple exit ba…

[deleted]

Re: Beyond automatic differentiation

#36

Post by godelski »

6gvONxR4sf7o wrote: post #27
modeless wrote: post #11

> We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). I'm not a huge fan of analyses like this that make one huge assumption that never holds in practice. Nobody uses full-batch gradient descent. It's not possible in practice or even desirable in theory. The stochasticity…

So what? This isn't some kind of gotcha. They're being explicit about the limitation. The built an interesting thing and showed that it does interesting things in a toy setting. This gets me excited about future work down this path.

With the hype many people have a difficult time interpreting research. Thinking it is all results oriented. For readers, you need to remember that there's still a lot of theory research that's important. Research is far more than getting benchmark results or flashy images. Benchmarks are only proxies for what we're actually trying to learn and as such need to be evaluated with care (I wrote about this recently). This is especially true for metics that use parametric models to evaluate, such as FID. It's problematic if we can't do research in a research field. Research isn't production, though the two can overlap.

Re: Beyond automatic differentiation

#37

Post by DougBTX »

modeless wrote: post #30
DougBTX wrote: post #25

Earlier quoted context omitted.

Batch size can be used for regularisation, but using it for that will limit training performance. From the Google Research Tuning Playbook: > The batch size governs the training speed and shouldn't be used to directly tune the validation set performance. Often, the ideal batch size will be the largest batch size supported by the available hardware. > […] > As long as all hyperparameters are well-tuned (especially the…

The source you cite is talking about "large" batch sizes which are still a miniscule fraction of the training set. Their (good) advice has no relevance at all in a discussion of full-batch training. Full-batch training is not the ideal case. It's a recipe for overfitting. We don't optimize models for their training set performance. We need them to generalize to the true data, of which our training set may not even be…

Take a look at the linked paper, they train using batches with about 5% of ImageNet in them (batch size = 2^16), which is pretty large from my perspective. Overall they show diminishing returns for larger batch sizes after a certain point. That fits the intuition that SGD is an approximation to GD, which gets better as batch size increases until the approximation error is negligible.
Post reply on HN