Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

241–250 of 269 posts

Re: Were RNNs all we needed?

#241

Post by roboboffin »

quantadev wrote: post #238
roboboffin wrote: post #236

Earlier quoted context omitted.

For example, papers like this call into question whether or not a LLM can plan: https://arxiv.org/html/2409.13373v1 This is a basic form of reasoning, to plan out the steps needed to execute something.

Planning, by definition, takes multiple reasoning steps. A single LLM inference is a fundamental single reasoning step, but it's a reasoning step nonetheless. It's like I'm saying a house is made of bricks. You can build a house of any shape out of bricks. But once bricks have been invented you can build houses. The LLM "reasoning" that even existed as early as GPT3.5 was the "brick" with which highly intelligent age…

It’s not reasoning, it retrieval of a pattern, and that pattern may contain reasoning.

The prompt engineering is the real reasoning, provided by the human.

Re: Were RNNs all we needed?

#242

Post by jszymborski »

jph00 wrote: post #231
jszymborski wrote: post #216

Earlier quoted context omitted.

> They were solved by LSTMs first proposed in 1997. I see this stuff everywhere online and it's often taught this way so I don't blame folks for repeating it, but I think it's likely promulgated by folks who don't train LSTMs with long contexts. LSTMs do add something like a "skip-connection" (before that term was a thing) which helps deal with the catastrophic vanishing gradients you get from e.g. Jordan RNNs right…

Highway networks add a skip connection, but LSTMs don't. Btw you might be interested in truncated backprop thru time, which we introduced in our ULMFiT paper.

I was referring to how the context vectors help avoid vanishing gradients by behaving very similarly to skip-connections, but yes, they aren't skip-connections as-such. That's been my understanding, at least.

We haven't tried truncated BPTT, but we certainly should.

Funnily enough, we adopted AWD-LSTMs, Ranger21, and Mish in the paper I linked after I heard about them through the fast.ai community (we also trialled QRNNs for a bit too). fast.ai has been hugely influential in my work.

Re: Were RNNs all we needed?

#243

Post by quantadev »

beckhamc wrote: post #234
quantadev wrote: post #222

Earlier quoted context omitted.

You can explain the "effect" of tanh at any level of abstraction you like, up to including describing things that happen in Semantic Space itself, but my description of what tanh is doing is 100% accurate in the context I used it. All it's doing is squashing a number down to below one. My understanding of how the Perceptron works is fully correct, and isn't missing any details. I've implemented many of them.

Your description of tanh isn't even correct, it squashes a real number to `(-1, 1)`, not "less than one". You're curious about whether there is gain in parameterising activation functions and learning them instead, or rather, why it's not used much in practice. That's an interesting and curious academic question, and it seems like you're already experimenting with trying out your own kinds of activation functions. Ho…

When discussing `tanh squashing` among other AI experts it's generally assumed that even the most pedantic and uncharitable parsing of words won't be able to misinterpret "smashing to less than one" as an incorrect sentence fragment, because the "one", in that context, obviously refers to distance from zero.

Re: Were RNNs all we needed?

#244

Post by kybernetikos »

seanhunter wrote: post #229
kybernetikos wrote: post #218

Earlier quoted context omitted.

> we don't want it to be like the brain and take 20 years or more to train? Estimates put training of gpt4 at something like 2500 gpu years to train, over about 10000 gpus. 20 years would be a big improvement.

1 GPU year is in no way comparable to 1 chronological year of learning for a human brain though.

Yes, but the underlying point is that in this case you can train the AI in parallel, and there's a decent chance this or something like it will be true for future AI architectures too. What does it matter that the AI needs to be trained on 20 years of experiences if all of those 20 years can be experienced in 6 months given the right hardware?

Re: Were RNNs all we needed?

#245

Post by quantadev »

roboboffin wrote: post #241
quantadev wrote: post #238

Earlier quoted context omitted.

Planning, by definition, takes multiple reasoning steps. A single LLM inference is a fundamental single reasoning step, but it's a reasoning step nonetheless. It's like I'm saying a house is made of bricks. You can build a house of any shape out of bricks. But once bricks have been invented you can build houses. The LLM "reasoning" that even existed as early as GPT3.5 was the "brick" with which highly intelligent age…

It’s not reasoning, it retrieval of a pattern, and that pattern may contain reasoning. The prompt engineering is the real reasoning, provided by the human.

Yeah, these kinds of discussions always devolve purely into debates about what's the proper definition of words. Especially on HN where everyone has their "Pedantic Knob" dialed up to 11.

Re: Were RNNs all we needed?

#246

Post by roboboffin »

quantadev wrote: post #245
roboboffin wrote: post #241

Earlier quoted context omitted.

It’s not reasoning, it retrieval of a pattern, and that pattern may contain reasoning. The prompt engineering is the real reasoning, provided by the human.

Yeah, these kinds of discussions always devolve purely into debates about what's the proper definition of words. Especially on HN where everyone has their "Pedantic Knob" dialed up to 11.

I understand your point. I apologise, if I am coming across pendantic.

My point is computers already follow algorithms, and algorithms contain reasoning; but the computers are not reasoning themselves. At least, not yet!

Re: Were RNNs all we needed?

#247

Post by hdivider »

jjmarr wrote: post #203
hdivider wrote: post #126

I still find it remarkable how we need such an extreme amount of electrical energy to power large modern AI models. Compare with one human brain. Far more sophisticated, even beyond our knowledge. What does it take to power it for a day? Some vegetables and rice. Still fine for a while if you supply pure junk food -- it'll still perform. Clearly we have a long, long way to go in terms of the energy efficiency of AI a…

Food is extremely dense in energy. 1 food calorie is about 1.1 Watt-hours. A hamburger is about 490 Wh. An AI model requires 0.047 kWh = 47 Wh to generate 1000 text responses.[1] If an LLM could convert hamburgers to energy, it could generate over 10000 prompt completions on a single hamburger. Based on my own experience, I would struggle to generate that much text without fries and a drink. [1] https://www.theverge.…

During that time, your brain would do far more than just that text generation though, beyond what we even know scientifically.

But yes, food energy could be useful for AI. A little dystopian potentially too, if you think about it. Like DARPA's EATR robot, able to run on plant biomass (although potentially animal biomass too, including human remains):

https://en.wikipedia.org/wiki/Energetically_Autonomous_Tacti...

Re: Were RNNs all we needed?

#248

Post by Legend2440 »

hdivider wrote: post #126

I still find it remarkable how we need such an extreme amount of electrical energy to power large modern AI models. Compare with one human brain. Far more sophisticated, even beyond our knowledge. What does it take to power it for a day? Some vegetables and rice. Still fine for a while if you supply pure junk food -- it'll still perform. Clearly we have a long, long way to go in terms of the energy efficiency of AI a…

This is more likely to be a hardware issue than an algorithms issue. The brain physically is a neural network, as opposed to a software simulation of one.

Re: Were RNNs all we needed?

#249

Post by visarga »

imjonse wrote: post #194
visarga wrote: post #184

Earlier quoted context omitted.

You could theoretically run the input twice, allowing the model to correlate later tokens with previous ones. It would fix the problem with not knowing what information to retain. A more complicated approach would train the RNN to request replaying some earlier data when needed. A great thing about RNNs is they can easily fork the state and generate trees, it would be possible to backtrack and work on combinatorial s…

Something like this? https://hazyresearch.stanford.edu/blog/2024-07-01-jrt

Yes, that's the paper.

Re: Were RNNs all we needed?

#250

Post by arolihas »

roboboffin wrote: post #246
quantadev wrote: post #245

Earlier quoted context omitted.

Yeah, these kinds of discussions always devolve purely into debates about what's the proper definition of words. Especially on HN where everyone has their "Pedantic Knob" dialed up to 11.

I understand your point. I apologise, if I am coming across pendantic. My point is computers already follow algorithms, and algorithms contain reasoning; but the computers are not reasoning themselves. At least, not yet!

You're not being pedantic at all. It's a crucial distinction that people try to wave away in favor of hype. Especially since we are so vulnerable to anthropomorphizing.
Post reply on HN