Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gradients are not all you need

arxiv.org

Post reply on HN
Print view View original

51–60 of 71 posts

Re: Gradients are not all you need

#51

Post by thatcherc »

Der_Einzige wrote: post #34

Global optimization techniques which don't rely on gradients seems theoretically superior in all instances, except that we haven't found super fast ways to run these kinds of optimizers. The cartpoll demo famously tripped up derivative based reinforcement learning for awhile.

Could you point me toward any more info or keywords on "the cartpoll demo famously tripped up derivative based reinforcement learning for awhile"? It sounds like an interesting bit of history but my google searches are only bringing up python tutorials.

Re: Gradients are not all you need

#52

Post by cultofmetatron »

mumblemumble wrote: post #24
sho_hn wrote: post #9

Earlier quoted context omitted.

Remember the days after Rowhammer and Heartbleed, when every new security vulnerability needed its own catchy name and domain name website? This is the science version of that. Branding is eating the world.

It's inescapable. 40 years ago, when articles went into print publications, you'd just get your paper into a key print journal and then trust that everyone who gets it would at least look through the article headlines and read the abstracts of articles that seemed relevant to them. And it was manageable because you'd only have a few new issues rolling in per month. But arXiv had an average of 167 CS papers being subm…

hmm. sounds like a good use case for the chatgpt api

Re: Gradients are not all you need

#53

Post by uoaei »

neurobama wrote: post #43
smonn_ wrote: post #29

Earlier quoted context omitted.

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.

>There's plenty of interesting neural network designs out there Where could a person learn more about these?

It's less about enumerating the architectures that have been tried before, and more about recognizing the modularity of NN components and the different perspectives on what those modules might represent.

Re: Gradients are not all you need

#55

Post by [deleted] »

neurobama wrote: post #43
smonn_ wrote: post #29

Earlier quoted context omitted.

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.

>There's plenty of interesting neural network designs out there Where could a person learn more about these?

[deleted]

Re: Gradients are not all you need

#56

Post by tangjurine »

thatcherc wrote: post #51
Der_Einzige wrote: post #34

Global optimization techniques which don't rely on gradients seems theoretically superior in all instances, except that we haven't found super fast ways to run these kinds of optimizers. The cartpoll demo famously tripped up derivative based reinforcement learning for awhile.

Could you point me toward any more info or keywords on "the cartpoll demo famously tripped up derivative based reinforcement learning for awhile"? It sounds like an interesting bit of history but my google searches are only bringing up python tutorials.

Chatgpt:

What does this refer to: cartpoll demo famously tripped up derivative based reinforcement learning

The phrase "cartpole demo famously tripped up derivative-based reinforcement learning" is likely referring to a classic problem in the field of reinforcement learning, which involves balancing a pole on a cart. The pole is attached to the cart via a hinge, and the goal is to keep the pole upright by moving the cart left or right in response to its angle. This problem is often used as a benchmark for testing reinforcement learning algorithms.

The phrase suggests that derivative-based reinforcement learning algorithms, which rely on computing gradients of a function with respect to its parameters, were not successful at solving this problem. This could be due to the fact that the problem is highly non-linear and requires precise control, which may be difficult to achieve with gradient-based methods.

Edit: bard got it too, with more detail, which is surprising

Re: Gradients are not all you need

#57

Post by msackmann »

thatcherc wrote: post #51
Der_Einzige wrote: post #34

Global optimization techniques which don't rely on gradients seems theoretically superior in all instances, except that we haven't found super fast ways to run these kinds of optimizers. The cartpoll demo famously tripped up derivative based reinforcement learning for awhile.

Could you point me toward any more info or keywords on "the cartpoll demo famously tripped up derivative based reinforcement learning for awhile"? It sounds like an interesting bit of history but my google searches are only bringing up python tutorials.

My guess: the cart pole is an inverted pendulum, and requires multiple left-right-swing-up movements to bring it from the “hanging” position to the “standing” position. Finding this action sequence using gradients of “where is the tip” vs “where should it be” is very hard, as swinging the pendulum to the left and right goes against the gradient.

Instead, using stochastic gradient approximations (policy gradient method such as proximal policy optimization) might be better suited to solving these kinds of problems. Effectively, they do not compute the exact gradient locally, but rather kind of a global approximation by trying out random sequences of actions and determining which of them are closest to the desired outcome.

Hence, stochastic gradient approximations might be considered some kind of hybrid between greedy local optimization (such as following the exact gradient) and global optimization.

Re: Gradients are not all you need

#58

Post by msackmann »

Interesting paper, thanks for bringing this up! I have been working on methods for trajectory optimization using both, analytic gradient computations and black box stochastic gradient approximations (proximal policy optimization).

I was always wondering about a question that is touched in the paper: despite the analytic gradient computation being intuitively more efficient and mathematically correct, it is much harder to learn a policy with it than with the “brute force trial-and-error” black box methods.

This paper brings many new perspectives on why.

Re: Gradients are not all you need

#59

Post by littlestymaar »

tangjurine wrote: post #56
thatcherc wrote: post #51

Earlier quoted context omitted.

Could you point me toward any more info or keywords on "the cartpoll demo famously tripped up derivative based reinforcement learning for awhile"? It sounds like an interesting bit of history but my google searches are only bringing up python tutorials.

Chatgpt: What does this refer to: cartpoll demo famously tripped up derivative based reinforcement learning The phrase "cartpole demo famously tripped up derivative-based reinforcement learning" is likely referring to a classic problem in the field of reinforcement learning, which involves balancing a pole on a cart. The pole is attached to the cart via a hinge, and the goal is to keep the pole upright by moving the…

You should be really careful with asking this kind of question to ChatGPT, because now you think you've learned the answer, but in fact there are two options with very different outcome:

- ChatGPT was trained on a corpus of data containing the answer and is able to give you a decent answer

- ChatGPT was never exposed to the answer and will hallucinate a plausible-sounding response, and because it will answer in a really convincing way, you'll get tricked into believing complete bullshit

Re: Gradients are not all you need

#60

Post by smonn_ »

kyllo wrote: post #46
smonn_ wrote: post #42

Earlier quoted context omitted.

I'm currently working on a variation of a spiking neural network that learns by making and purging connections between neurons, which so far has been pretty interesting, though I am having a hard time getting it to output anything more than just the patterns it recognised. I did play around with adding its outputs to the input list, making it sort of recurrent but its practically impossible to decode anything thats g…

Are you familiar with the edge popup algorithm introduced in "What's Hidden in a Randomly Weighted Neural Network?" https://arxiv.org/abs/1911.13299v2 Seems relevant to what you're working on. It starts with a randomly initialized, overparameterized neural net, but instead of gradient descent backpropagation, it learns by deleting connection edges.

I haven't read it, thanks a lot! I'm probably going to use it in an essay I'm writing about the topic.
Post reply on HN