Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

201–210 of 269 posts

Re: Were RNNs all we needed?

#201

Post by uh_uh »

quantadev wrote: post #140
uh_uh wrote: post #117

Earlier quoted context omitted.

> That's the only non-linearity I'm aware of. "only" is doing a lot work here because that non-linearity is enough to vastly expand the landscape of functions that an NN can approximate. If the NN was linear, you could greatly simplify the computational needs of the whole thing (as was implied by another commenter above) but you'd also not get a GPT out of it.

All the trainable parameters are just slopes of lines tho. Training NNs doesn't involve adjusting any inputs to non-linear functions. The tanh smashing function just makes sure nothing can blow up into large numbers and all outputs are in a range of less than 1. There's no "magic" or "knowledge" in the tanh smashing. All the magic is 100% in the weights. They're all linear. The amazing thing is that all weights are l…

> The tanh smashing function just makes sure nothing can blow up into large numbers and all outputs are in a range of less than 1.

That's not the main point even though it probably helps. As OkayPhysicist said above, without a nonlinearity, you could collapse all the weight matrices into a single matrix. If you have 2 layers (same size, for simplicity) described by weight matrices A and B, you could multiply them and get C, which you could use for inference.

Now, you can do this same trick not only with 2 layers but 100 million, all collapsing into a single matrix after multiplication. If the nonlinearities weren't there, the effective information content of the whole NN would collapse into that of a single-layer NN.

Re: Were RNNs all we needed?

#202

Post by aDyslecticCrow »

charlescurt123 wrote: post #83

I find the entire field lacking when it comes to long-horizon problems. Our current, widely used solution is to scale, but we're nowhere near achieving the horizon scales even small mammal brains can handle. Our models can have trillions of parameters, yet a mouse brain would still outperform them on long-horizon tasks and efficiency. It's something small, simple, and elegant—an incredible search algorithm that not o…

Yes... The field lacks the HOLY GRAIL (long-horizon problems). But we don't need a mouse-brain to sort spam emails. The Hail Mary 2B+ parameter models and above are still niche uses of these algorithms (too heavy to run practically). There is plenty of room for clever and small models running on limited hardware and datasets to solve useful problems and nothing more.

Models that change size as needed have been experimented with, but they are either too inefficient or difficult to optimize at a limited power budget. However, I agree that they are likely what is needed if we want to continue to scale upward in size.

I suspect the real bottleneck is a breakthrough in training itself. Backpropagation loss is too simplistic to optimize our current models perfectly, let alone future larger ones. But there is no guarantee a better alternative exists which may create a fixed limit to current ML approaches.

Re: Were RNNs all we needed?

#203

Post by jjmarr »

hdivider wrote: post #126

I still find it remarkable how we need such an extreme amount of electrical energy to power large modern AI models. Compare with one human brain. Far more sophisticated, even beyond our knowledge. What does it take to power it for a day? Some vegetables and rice. Still fine for a while if you supply pure junk food -- it'll still perform. Clearly we have a long, long way to go in terms of the energy efficiency of AI a…

Food is extremely dense in energy. 1 food calorie is about 1.1 Watt-hours. A hamburger is about 490 Wh. An AI model requires 0.047 kWh = 47 Wh to generate 1000 text responses.[1] If an LLM could convert hamburgers to energy, it could generate over 10000 prompt completions on a single hamburger.

Based on my own experience, I would struggle to generate that much text without fries and a drink.

[1] https://www.theverge.com/24066646/ai-electricity-energy-watt...

Re: Were RNNs all we needed?

#204

Post by chasd00 »

mafribe wrote: post #198
wslh wrote: post #193

Earlier quoted context omitted.

ELI5: Could you explain what neuromorphic approaches mean, and how they contribute to AI/AGI? My first impression as a layperson (probably wrong) is that this approach resembles ideas from the book "The Society of the Mind", where the system isn't just simulating neurons but involves a variety of methods and interactions across "agents" or sub-systems.

Neuromorphic has been an ongoing failure (for general purpose processors or even AI accelerators), ever since Carver Mead introduced (and quickly abandoned them) them nearly half a century ago. Bill Dally (NVidia CTO) concurs: "I keep getting those calls from those people who claim they are doing neuromorphic computing and they claim there is something magical about it because it's the way that the brain works ... bu…

I like the feather analogy. Early on all humans knew about flight was from biology (watching birds fly) but trying to make a flying machine modeled after a bird would never work. We can fly today but plane designs are nothing like biological flying machines. In the same way, all we know about intelligence comes from biology and trying to invent an AGI modeled on biological intelligence may be just as impossible as a plane designed around how birds fly.

/way out of my area of expertise here

Re: Were RNNs all we needed?

#205

Post by aDyslecticCrow »

mkaic wrote: post #35

I strongly enjoy the simplicity of their "minGRU" architecture. It's basically just: class MinGRU(nn.Module): def __init__(self, token_size, hidden_state_size): self.token_to_proposal = nn.Linear(token_size, hidden_size) self.token_to_mix_factors = nn.Linear(token_size, hidden_size) def forward(self, previous_hidden_state, current_token): proposed_hidden_state = self.token_to_proposal(current_token) mix_factors = tor…

I don't think it's a capital-B Breakthrough, but recurrent networks are everywhere, and a simplification that improved training and performance clears the stage to build back complexity up again to even higher hights.

Re: Were RNNs all we needed?

#206

Post by seanhunter »

bob1029 wrote: post #196
wslh wrote: post #193

Earlier quoted context omitted.

ELI5: Could you explain what neuromorphic approaches mean, and how they contribute to AI/AGI? My first impression as a layperson (probably wrong) is that this approach resembles ideas from the book "The Society of the Mind", where the system isn't just simulating neurons but involves a variety of methods and interactions across "agents" or sub-systems.

Neuromorphic mostly just means "like how the brain works". It encompasses a variety of software & hardware approaches. The most compelling and obvious one to me is hardware purpose-built to simulate spiking neural networks. In the happy case, SNNs are extremely efficient. Basically consuming no energy. You could fool yourself into thinking we can just do this on the CPU due to the sparsity of activations. I think the…

We somehow want a network that is neuromorphic in structure but we don't want it to be like the brain and take 20 years or more to train?

Secondly how do we get to claim that a particular thing is neuromorphic when we have such a rudimentary understanding of how a biological brain works or how it generates things like a model of the world, understanding of self etc etc.

Re: Were RNNs all we needed?

#207

Post by falcor84 »

ithkuil wrote: post #189
falcor84 wrote: post #175

Earlier quoted context omitted.

Would you mind expanding upon your thesis? If that compute and all those parameters aren't "fitting" the training examples, what is it that the model is learning, and how should that be analyzed?

I think there are two distinct areas. One is the building of the representations, which is achieved by fitting. The other area is loosely defined as "computing" which is some kind of searching for a path through representation space. All of that is wrapped in a translation layer that can turn those representations into stuff we humans can understand and interact with. All of that is achieved to some extent by current…

But how does it get good at "computing"? The way I see it, we either program them to do so manually, or we use ML, at which case the model "fits" the computation based on training examples or environmental feedback, no? What am I missing?

Re: Were RNNs all we needed?

#208

Post by manjunaths »

bob1029 wrote: post #22

> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually. > RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction. I would like to draw an analogy to digital signal processing. If you think of the recur…

Can we even implement IIR filters to give good performance and scaling at large scale on current architectures like GPUs ?

Re: Were RNNs all we needed?

#209

Post by fennecfoxy »

wslh wrote: post #193
bob1029 wrote: post #22

> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually. > RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction. I would like to draw an analogy to digital signal processing. If you think of the recur…

ELI5: Could you explain what neuromorphic approaches mean, and how they contribute to AI/AGI? My first impression as a layperson (probably wrong) is that this approach resembles ideas from the book "The Society of the Mind", where the system isn't just simulating neurons but involves a variety of methods and interactions across "agents" or sub-systems.

I love this book and have it sitting on my shelf right now! Read it when I was a kid and was amazed at the ideas in it, nowadays it's clearer to me that the author only had a grasp of how things like that would be built but still cool nonetheless.

I would highly recommend it to people who love a good "near future" scifi book.

Re: Were RNNs all we needed?

#210

Post by fennecfoxy »

mvkel wrote: post #144
foota wrote: post #93

Earlier quoted context omitted.

Not even then, a truly fully connected network would have super exponential runtime (it would take N^N time to evaluate)

Wetware is the future.

Can't wait to see this defiantly spray painted across a torn up brick wall while computronium brained super intelligences slowly disassemble our planet to make paperclips.
Post reply on HN