Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

251–260 of 269 posts

Re: Were RNNs all we needed?

#251

Post by quantadev »

roboboffin wrote: post #246
quantadev wrote: post #245

Earlier quoted context omitted.

Yeah, these kinds of discussions always devolve purely into debates about what's the proper definition of words. Especially on HN where everyone has their "Pedantic Knob" dialed up to 11.

I understand your point. I apologise, if I am coming across pendantic. My point is computers already follow algorithms, and algorithms contain reasoning; but the computers are not reasoning themselves. At least, not yet!

You weren't being pedantic yourself. My point is that this discussion is ultimately about the definition of words, and that all by itself, makes the discussion meaningless.

I think a "granule" of "reasoning" happens at each inference, and you think there is no reasoning in a single inference. To discuss it further would be a game of whose definition of any given word is correct.

Re: Were RNNs all we needed?

#252

Post by mickg10 »

quantadev wrote: post #235
wrs wrote: post #228

Earlier quoted context omitted.

Well. The word “linear” the way you use it doesn’t seem to have any particular meaning, certainly not the standard mathematical meaning, so I’m not sure we can make further progress on this explanation. I’ll just reiterate that the single “technical” (whatever that means) nonlinearity in ReLU is exactly what lets a layer approximate any continuous[*] function. [*] May have forgotten some more adjectives here needed f…

If you're confused just show a tanh graph and a ReLU graph to a 7 year old child and ask which one is linear. They'll all get it right. So you're not confused in the slightest bit about anything I've said. There's nothing even slightly confusing about saying a ReLU is made of two lines.

I.e. ReLU is _piecewise_ linear. That discontinuity that separates the 2 pieces is precisely what makes it non linear. Which is what enables the actual universal approximation.

Re: Were RNNs all we needed?

#253

Post by jjmarr »

hdivider wrote: post #247
jjmarr wrote: post #203

Earlier quoted context omitted.

Food is extremely dense in energy. 1 food calorie is about 1.1 Watt-hours. A hamburger is about 490 Wh. An AI model requires 0.047 kWh = 47 Wh to generate 1000 text responses.[1] If an LLM could convert hamburgers to energy, it could generate over 10000 prompt completions on a single hamburger. Based on my own experience, I would struggle to generate that much text without fries and a drink. [1] https://www.theverge.…

During that time, your brain would do far more than just that text generation though, beyond what we even know scientifically. But yes, food energy could be useful for AI. A little dystopian potentially too, if you think about it. Like DARPA's EATR robot, able to run on plant biomass (although potentially animal biomass too, including human remains): https://en.wikipedia.org/wiki/Energetically_Autonomous_Tacti...

AI is more energy-efficient than a human doing the same language-generation task is my point.

Re: Were RNNs all we needed?

#254

Post by lr1970 »

bob1029 wrote: post #22

> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually. > RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction. I would like to draw an analogy to digital signal processing. If you think of the recur…

Again from signal processing: depending on position of the poles in z-transformed filter transfer function the output of IIR has a narrow stability region that is typically carefully designed for. Otherwise IIR filters either exponentially decay to zero to exponentially grow to infinity. RNN cells like LSTM are "decaying filters" with non-linear gates introduced to stop decay and to "remember" things.

FIR filters are way simpler to design and can capture memory without hacks.

Re: Were RNNs all we needed?

#255

Post by quantadev »

mickg10 wrote: post #252
quantadev wrote: post #235

Earlier quoted context omitted.

If you're confused just show a tanh graph and a ReLU graph to a 7 year old child and ask which one is linear. They'll all get it right. So you're not confused in the slightest bit about anything I've said. There's nothing even slightly confusing about saying a ReLU is made of two lines.

I.e. ReLU is _piecewise_ linear. That discontinuity that separates the 2 pieces is precisely what makes it non linear. Which is what enables the actual universal approximation.

Which is what I said two replies ago.

Followed by "in some sense it's [ReLU] still even MORE linear than tanh or sigmoid functions are". There's no way you misunderstood that sentence, or took it as my "definition" of linearity...so I guess you just wanted to reaffirm I was correct, again, so thanks.

Re: Were RNNs all we needed?

#256

Post by quantadev »

quantadev wrote: post #151
mr_toad wrote: post #112

Earlier quoted context omitted.

> It's astounding that we DID get the emergent intelligence from just doing this "curve fitting" onto "lines" rather than actual "curves". In Ye Olden days (the 90’s) we used to approximate non-linear models using splines or seperate slopes models - fit by hand. They were still linear, but with the right choice of splines you could approximate a non-linear model to whatever degree of accuracy you wanted. Neural netwo…

In college (BSME) I wrote a computer program to generate cam profiles from Bezier curves. It's just a programming trick to generate curves from straight lines at any level of accuracy you want just by letting the computer take smaller and smaller steps. It's an interesting concept to think of how NNs might be able to exploit this effect in some way based on straight lines in the weights, because a very small number o…

In fact now that I think about it, for any 3 or more points in Semantic Space, there would necessarily be a "Bezier Path" which would have genuine meaning at every point as a good smooth differentiable path thru higher dimensional space to get from one point to another point while "visiting" all intermediate other points. This has to have a direct use in LLMs in terms of reasoning.

Re: Were RNNs all we needed?

#257

Post by YeGoblynQueenne »

jszymborski wrote: post #216
YeGoblynQueenne wrote: post #191

Earlier quoted context omitted.

Vanishing (or exploding) gradients affected all deep architectures, not just RNNs. They were solved by LSTMs first proposed in 1997. See: https://www.semanticscholar.org/paper/Long-Short-Term-Memory... I find it interesting that this knowledge seems to be all but forgotten now. Back in the day, ca. 2014, LSTMs were all the rage, e.g. see: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.i…

> They were solved by LSTMs first proposed in 1997. I see this stuff everywhere online and it's often taught this way so I don't blame folks for repeating it, but I think it's likely promulgated by folks who don't train LSTMs with long contexts. LSTMs do add something like a "skip-connection" (before that term was a thing) which helps deal with the catastrophic vanishing gradients you get from e.g. Jordan RNNs right…

>> I see this stuff everywhere online and it's often taught this way so I don't blame folks for repeating it, but I think it's likely promulgated by folks who don't train LSTMs with long contexts.

To clarify, this wasn't taught to me. I studied LSTMs during my MSc in 2014, by my own initiative, because they were popular at the time [1]. I remember there being a hefty amount of literature on LSTMs, and I mean scholarly articles, not just blog posts. Rather at the time I think there were only two blog posts, the ones by Andrey Karpathy and Chris Olah that I link above. The motivation with respect to vanishing gradients is well documented in previous wok by Hochreiter (I think it's his thesis), and maybe a little less so in the 1997 paper that introduces the "constant error carousel".

What kind of "instability" did you see? Vanishing gradients weren't something I noticed in my experiments. If that was because I didn't use a long enough context, as you say, I wouldn't be able to tell but there was a different kind of instability: loss would enter an oscillatory pattern which I put down to the usual behaviour of gradient descent (either it gets stuck on local minima, or in saddle points). Is that what you mean?

_______________

[1] More precisely, our tutor asked us to study an RNN architecture expecting we'd look at something relatively simple like an Elman network but I wanted to try out the hot new stuff. The code and report is here:

https://github.com/stassa/lstm_rnn

There may be errors in the code and I don't know if you'll be able to run it, in case you got really curious. I don't think I really grokked automatic differentiation at the time.

Re: Were RNNs all we needed?

#258

Post by ActorNightly »

bob1029 wrote: post #154
spencerchubb wrote: post #134

Earlier quoted context omitted.

When you train a neural network, it is not search, it is descending through a curve. If you were to search for billions of parameters by brute force, you literally could not do it in the lifespan of the universe. A neural network is differentiable, meaning you can take the derivative of it. You train the parameters by taking finding gradient with respect to each parameter, and going in the opposite direction. Hence t…

A biological neural network is certainly not differentiable. If the thing we want to build is not realizable with this technique, why can't we move on from it? Gradient descent isn't the only way to do this. Evolutionary techniques can explore impossibly large, non-linear problem spaces. Being able to define any kind of fitness function you want is sort of like a super power. You don't have to think in such constrain…

The issue is that its still a massive search space.

You can do this yourself, go play nandgame, and beat it, at which point you should be able to make a cpu out of nandgates. Then set up a rnn that is the same layers at total layers of the nandgates and as wide as all the inputs, with every output being fed back into the first input. Then do PSO or GA on all the weights and see how long it takes you to make a fully functioning cpu.

Re: Were RNNs all we needed?

#259

Post by shaklee3 »

manjunaths wrote: post #208
bob1029 wrote: post #22

> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually. > RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction. I would like to draw an analogy to digital signal processing. If you think of the recur…

Can we even implement IIR filters to give good performance and scaling at large scale on current architectures like GPUs ?

Yes. See this paper: http://cs.txstate.edu/~mb92/papers/asplos18.pdf

And things have improved a lot since then.

Re: Were RNNs all we needed?

#260

Post by shaklee3 »

bob1029 wrote: post #227
manjunaths wrote: post #208

Earlier quoted context omitted.

Can we even implement IIR filters to give good performance and scaling at large scale on current architectures like GPUs ?

I don't think so. FIR filters can be unrolled and parallelized over the data. These are definitely possible to do on GPU to great effect. But, IIR filters constantly depend on the output of the prior time step, so you can't unroll anything. These would probably be faster to simulate on the CPU.

See my comment above. It's definitely doable and very fast.
Post reply on HN