Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradients are not all you need

arxiv.org

Post reply on HN
Print view View original

31–40 of 71 posts

Re: Gradients are not all you need

#31

Post by alephxyz »

hikingsimulator wrote: post #4
unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

There is a lot of jargon in ML, an example is found in the object detection literature where you will often find sentences like such: "we present a novel simple non-hierarchical feature network backbone with window-shifted attention based learning accommodating neck+regression head fine-tuning or masked cascade RCNN second stage." I'm half joking. Surveys are often a godsend.

You forgot to toss in a few "state-of-the-art"

Re: Gradients are not all you need

#32

Post by tough »

Bootvis wrote: post #30
tough wrote: post #26

Earlier quoted context omitted.

I asked GPT > https://arxiv.org/abs/2111.05803 make a good useful title that's not silly for this paper and wht's about I dunno if it even read the article lols ==== Title: “Deep Reinforcement Learning for Multi-Agent Navigation in Complex Environments” This paper proposes a deep reinforcement learning approach for multi-agent navigation in complex environments. The proposed method uses a centralized critic and decen…

ChatGPT doesn’t read URL’s on the fly but it will pretend it does.

lol

Re: Gradients are not all you need

#33

Post by ajb »

unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

There's quite a long history of these titles, even before ML:

"Sometime’ is Sometimes ‘Not Never’" - Lamport

"Reflections on trusting trust" - Thompson

"On the cruelty of really teaching computer science" - Dijkstra

I'm sure there are more

Re: Gradients are not all you need

#34

Post by Der_Einzige »

Global optimization techniques which don't rely on gradients seems theoretically superior in all instances, except that we haven't found super fast ways to run these kinds of optimizers.

The cartpoll demo famously tripped up derivative based reinforcement learning for awhile.

Re: Gradients are not all you need

#35

Post by visarga »

xpe wrote: post #27
visarga wrote: post #7

Earlier quoted context omitted.

> "Understanding Limitations and Chaos-Based Failures in Gradient-Based Optimization Methods." (gpt4) Fixed with ML.

I’ve provided the PDF URL to ChatGPT 4.0 and asked it to summarize the article with alternative titles, but for some reason it keeps getting the original title, authors, abstract, and body wrong. What prompt did you use?

I asked a bunch of things. So initially I posted the abstract and title, then prompted just: "Explain". Then I asked "Give me background knowledge on spectrum of the Jacobian" and "Explain the title of the paper" and in the end "Reformulate the title in a more explicit manner". Maybe you can skip directly to the last prompt.

Re: Gradients are not all you need

#36

Post by wenc »

Der_Einzige wrote: post #34

Global optimization techniques which don't rely on gradients seems theoretically superior in all instances, except that we haven't found super fast ways to run these kinds of optimizers. The cartpoll demo famously tripped up derivative based reinforcement learning for awhile.

> Global optimization techniques which don't rely on gradients seems theoretically superior in all instances, except that we haven't found super fast ways to run these kinds of optimizers.

Did you mean "Global optimization techniques which do rely on gradients..."? Because exact gradient-based global optimization (GBD or branch-and-bound based) methods for general nonconvex problems are theoretically superior (bounding with McCormick relaxations etc.) but also more challenging to practically deploy than say stochastic methods or metaheuristics like local search.

Re: Gradients are not all you need

#37

Post by abhgh »

unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

Your comment reminded me this talk [1] by David Duvenaud where he discusses his work on Neural ODEs, introduced in the paper [2] titled "Neural Ordinary Differential Equations" (this was one of the Best Papers in Neurips 2018). Apparently the original name of the paper was "Training Infinitesimally-Layered Neural Networks by Backpropagating through Black-box ODE Solvers". I found that to be much better description for the paper. It seems he thinks so too, but eventually the catch-phrase-y title won out. In the video [1], this bit is at 00:48-01:22.

PS: [1] is a good talk in general where he discusses some of the limitations of the paper and things that could have been done better.

[1] https://www.youtube.com/watch?v=YZ-_E7A3V2w

[2] https://papers.nips.cc/paper_files/paper/2018/file/69386f6bb...

Re: Gradients are not all you need

#38

Post by wcoenen »

xpe wrote: post #27
visarga wrote: post #7

Earlier quoted context omitted.

> "Understanding Limitations and Chaos-Based Failures in Gradient-Based Optimization Methods." (gpt4) Fixed with ML.

I’ve provided the PDF URL to ChatGPT 4.0 and asked it to summarize the article with alternative titles, but for some reason it keeps getting the original title, authors, abstract, and body wrong. What prompt did you use?

That's because chatgpt doesn't have the ability to retrieve PDFs from the internet. (Unless maybe if you have early access to the version with plugins?)

Bing chat does have the ability to read opened PDFs when used from the Edge side bar.

Re: Gradients are not all you need

#39

Post by chas »

hikingsimulator wrote: post #4
unlikelymordant wrote: post #2

My one wish is that machine learning papers would use paper titles that actually described what the paper was about. I suppose there is a certain 'evolutionary pressure' where clever titles 'outcompete' dryer, more descriptive titles (or it seems that way). But i don't like it.

There is a lot of jargon in ML, an example is found in the object detection literature where you will often find sentences like such: "we present a novel simple non-hierarchical feature network backbone with window-shifted attention based learning accommodating neck+regression head fine-tuning or masked cascade RCNN second stage." I'm half joking. Surveys are often a godsend.

The major ML conferences all have pretty tight page limits, so more expository sentences usually get cut. This also means that papers usually only explain how their work is different from previous work, so they assume you are familiar with the papers they cite or are willing to read the cited papers.

This means that people who have an up-to-date knowledge of a given subfield can quickly get a lot out of a new papers. Unfortunately, it also means that it usually takes a pretty decent stack of papers to get up to speed on a new subfield since you have to read the important segments of the commonly cited papers in order to gain the common knowledge that papers are being diffed against.

Traditionally, this issue is solved by textbooks, since the base set of ideas in a given field or subfield is pretty stable. ML has been moving pretty fast in recent years, so there is still a sizable gap between the base knowledge required for productive paper reading and what you can get out of a textbook. For example, Goodfellow et al [1] is a great intro to the core ideas of deep learning, but it was published before transformers were invented, so it doesn’t mention them at all.

[1] https://www.deeplearningbook.org/

Re: Gradients are not all you need

#40

Post by mumblemumble »

smonn_ wrote: post #29
modeless wrote: post #17

Seems to me like the whole history of neural nets is basically crafting models with well-behaved gradients to make gradient descent work well. That, and models that can achieve high utilization of available hardware. The surprising thing is that models exist where the gradients are so well-behaved that we can learn GPT-4 level stuff.

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.

Also, multi-head attention strikes me as being about as close to how language semantics seems to actually work in human brains as I've seen.

Lots of caveats there, of course. First off, I don't know much about the neurology, I just have an amateur interest in second language acquisition research that sometimes brings me into contact with this sort of thing. On the ANN side, which is closer to my actual wheelhouse, we definitely don't actually have any way of knowing if the actual mechanism is all that close, and I'm guessing it probably isn't even close since ANN's don't actually work that similarly to brains. Nor does it need to be, but, intuitively, there's still something promising about an ANN architecture that's vaguely capable of mimicking the behavior of modules in an existing system (human brains) that's well known to be capable of doing the job. I'm not super wild about the bidirectional recurrent layers, either, because they impose some restrictions that clearly aren't great, such as the hard limit on input size. et cetera. But it still strikes me as another big step in a good direction.

Post reply on HN