Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

231–240 of 269 posts

Re: Were RNNs all we needed?

#231

Post by jph00 »

jszymborski wrote: post #216
YeGoblynQueenne wrote: post #191

Earlier quoted context omitted.

Vanishing (or exploding) gradients affected all deep architectures, not just RNNs. They were solved by LSTMs first proposed in 1997. See: https://www.semanticscholar.org/paper/Long-Short-Term-Memory... I find it interesting that this knowledge seems to be all but forgotten now. Back in the day, ca. 2014, LSTMs were all the rage, e.g. see: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.i…

> They were solved by LSTMs first proposed in 1997. I see this stuff everywhere online and it's often taught this way so I don't blame folks for repeating it, but I think it's likely promulgated by folks who don't train LSTMs with long contexts. LSTMs do add something like a "skip-connection" (before that term was a thing) which helps deal with the catastrophic vanishing gradients you get from e.g. Jordan RNNs right…

Highway networks add a skip connection, but LSTMs don't. Btw you might be interested in truncated backprop thru time, which we introduced in our ULMFiT paper.

Re: Were RNNs all we needed?

#232

Post by WithinReason »

immibis wrote: post #230
WithinReason wrote: post #213

Earlier quoted context omitted.

no, you can give as much context to a transformer as you want, you just run out of memory

An RNN doesn't run out of memory from that, so they are still fundamentally different. How do you encode arbitrarily long positions, anyway?

They are different but transformers don't have fixed windows, you can extend the context or make it smaller. I think you can extend a positional encoding if it's not a learned encoding.

Re: Were RNNs all we needed?

#234

Post by beckhamc »

quantadev wrote: post #222
uh_uh wrote: post #201

Earlier quoted context omitted.

> The tanh smashing function just makes sure nothing can blow up into large numbers and all outputs are in a range of less than 1. That's not the main point even though it probably helps. As OkayPhysicist said above, without a nonlinearity, you could collapse all the weight matrices into a single matrix. If you have 2 layers (same size, for simplicity) described by weight matrices A and B, you could multiply them and…

You can explain the "effect" of tanh at any level of abstraction you like, up to including describing things that happen in Semantic Space itself, but my description of what tanh is doing is 100% accurate in the context I used it. All it's doing is squashing a number down to below one. My understanding of how the Perceptron works is fully correct, and isn't missing any details. I've implemented many of them.

Your description of tanh isn't even correct, it squashes a real number to `(-1, 1)`, not "less than one".

You're curious about whether there is gain in parameterising activation functions and learning them instead, or rather, why it's not used much in practice. That's an interesting and curious academic question, and it seems like you're already experimenting with trying out your own kinds of activation functions. However, people in this thread (including myself) wanted to clarify some perceived misunderstandings you had about nonlinearities and "why" they are used in DNNs. Or how "squashing functions" is a misnomer because `g(x) = x/1000` doesn't introduce any nonlinearities. Yet you continue to fixate and double down on your knowledge of "what" a tanh is, and even that is incorrect.

Re: Were RNNs all we needed?

#235

Post by quantadev »

wrs wrote: post #228
quantadev wrote: post #149

Earlier quoted context omitted.

ReLU technically has a non-linearity at zero, but in some sense it's still even MORE linear than tanh or sigmoid, so it just demonstrates even better than tanh-type squashing that all this LLM stuff is being done ultimately with straight line math. All a ReLU function does is choose which line to use, a sloped one or a zero one.

Well. The word “linear” the way you use it doesn’t seem to have any particular meaning, certainly not the standard mathematical meaning, so I’m not sure we can make further progress on this explanation. I’ll just reiterate that the single “technical” (whatever that means) nonlinearity in ReLU is exactly what lets a layer approximate any continuous[*] function. [*] May have forgotten some more adjectives here needed f…

If you're confused just show a tanh graph and a ReLU graph to a 7 year old child and ask which one is linear. They'll all get it right. So you're not confused in the slightest bit about anything I've said. There's nothing even slightly confusing about saying a ReLU is made of two lines.

Re: Were RNNs all we needed?

#236

Post by roboboffin »

quantadev wrote: post #220
roboboffin wrote: post #188

Earlier quoted context omitted.

I'm not sure that's true at all. There are several well known researchers that say LLMs are in fact not doing reasoning.

Those are all the people that have not yet decoupled "reasoning" from "consciousness" in their own way of thinking. It's admittedly hyperbolic to say "everyone". I love hyperbole on HN. :)

For example, papers like this call into question whether or not a LLM can plan:

https://arxiv.org/html/2409.13373v1

This is a basic form of reasoning, to plan out the steps needed to execute something.

Re: Were RNNs all we needed?

#237

Post by radarsat1 »

ants_everywhere wrote: post #40
vundercind wrote: post #26

Earlier quoted context omitted.

I eagerly await the backlash to suggesting any one thing is all you need, the first shot of which shall surely be titled: “‘All you need’ Considered Harmful”

Surely the universe is all you need though

Interstellar taught me that love transcends the universe. Ergo..

Re: Were RNNs all we needed?

#238

Post by quantadev »

roboboffin wrote: post #236
quantadev wrote: post #220

Earlier quoted context omitted.

Those are all the people that have not yet decoupled "reasoning" from "consciousness" in their own way of thinking. It's admittedly hyperbolic to say "everyone". I love hyperbole on HN. :)

For example, papers like this call into question whether or not a LLM can plan: https://arxiv.org/html/2409.13373v1 This is a basic form of reasoning, to plan out the steps needed to execute something.

Planning, by definition, takes multiple reasoning steps. A single LLM inference is a fundamental single reasoning step, but it's a reasoning step nonetheless.

It's like I'm saying a house is made of bricks. You can build a house of any shape out of bricks. But once bricks have been invented you can build houses. The LLM "reasoning" that even existed as early as GPT3.5 was the "brick" with which highly intelligent agents can be built out of, with no further "breakthroughs" being required.

The basic Transformer Architecture was enough and already has the magical ingredient of reasoning. The rest is just a matter of prompt engineering.

Re: Were RNNs all we needed?

#239

Post by [deleted] »

jph00 wrote: post #231
jszymborski wrote: post #216

Earlier quoted context omitted.

> They were solved by LSTMs first proposed in 1997. I see this stuff everywhere online and it's often taught this way so I don't blame folks for repeating it, but I think it's likely promulgated by folks who don't train LSTMs with long contexts. LSTMs do add something like a "skip-connection" (before that term was a thing) which helps deal with the catastrophic vanishing gradients you get from e.g. Jordan RNNs right…

Highway networks add a skip connection, but LSTMs don't. Btw you might be interested in truncated backprop thru time, which we introduced in our ULMFiT paper.

[deleted]

Re: Were RNNs all we needed?

#240

Post by twobitshifter »

YeGoblynQueenne wrote: post #191
theanonymousone wrote: post #158

I remember that, the way I understood it, Transformers solved two major "issues" of RNNs that enabled the later boom: Vanishing gradients limiting the context (and model?) size and difficulty in parallelisation limiting the size of the training data. Do we have solutions for these two problems now?

Vanishing (or exploding) gradients affected all deep architectures, not just RNNs. They were solved by LSTMs first proposed in 1997. See: https://www.semanticscholar.org/paper/Long-Short-Term-Memory... I find it interesting that this knowledge seems to be all but forgotten now. Back in the day, ca. 2014, LSTMs were all the rage, e.g. see: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.i…

Agreed, Ilya Sutskever himself has spent a long time with lstm and published papers like this one while working at Google. http://proceedings.mlr.press/v37/jozefowicz15.pdf

Recent comments from him have said that any architecture can achieve transformer accuracy and recall, but we have devoted energy to refining transformers, due to the early successes.

Post reply on HN