Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

211–220 of 269 posts

Re: Were RNNs all we needed?

#211

Post by fennecfoxy »

observationist wrote: post #61
jprete wrote: post #55

Earlier quoted context omitted.

I wonder if there's something about tech culture - or tech people - that encourages them to really, really like snowclones.

Yes. Do stuff that other people have been successful doing. Monkey see, monkey do - it's not a tech people thing, it's a human thing. Tech just happens to be most on display at the moment - because tech people are building the tools and the parameters and the infrastructure handling all our interactions.

Not sure why people are surprised about this when it's the modus operandi of all life on the planet.

I could spam we are the stochastic parrots after all, yet one more time.

Re: Were RNNs all we needed?

#212

Post by ithkuil »

falcor84 wrote: post #207
ithkuil wrote: post #189

Earlier quoted context omitted.

I think there are two distinct areas. One is the building of the representations, which is achieved by fitting. The other area is loosely defined as "computing" which is some kind of searching for a path through representation space. All of that is wrapped in a translation layer that can turn those representations into stuff we humans can understand and interact with. All of that is achieved to some extent by current…

But how does it get good at "computing"? The way I see it, we either program them to do so manually, or we use ML, at which case the model "fits" the computation based on training examples or environmental feedback, no? What am I missing?

the distinction is fuzzy indeed, especially if any thing that you "program in manually" has some parameters that are learned.

Conceptually we already have parts of the model that are not learned: the architecture of the model itself.

Re: Were RNNs all we needed?

#213

Post by WithinReason »

immibis wrote: post #54
dheera wrote: post #53

Earlier quoted context omitted.

I mean, transformer-based LLMs are RNNs, just really really really big ones with very wide inputs that maintain large amounts of context.

No. An RNN has an arbitrarily-long path from old inputs to new outputs, even if in practice it can't exploit that path. Transformers have fixed-size input windows.

no, you can give as much context to a transformer as you want, you just run out of memory

Re: Were RNNs all we needed?

#214

Post by wslh »

bob1029 wrote: post #196
wslh wrote: post #193

Earlier quoted context omitted.

ELI5: Could you explain what neuromorphic approaches mean, and how they contribute to AI/AGI? My first impression as a layperson (probably wrong) is that this approach resembles ideas from the book "The Society of the Mind", where the system isn't just simulating neurons but involves a variety of methods and interactions across "agents" or sub-systems.

Neuromorphic mostly just means "like how the brain works". It encompasses a variety of software & hardware approaches. The most compelling and obvious one to me is hardware purpose-built to simulate spiking neural networks. In the happy case, SNNs are extremely efficient. Basically consuming no energy. You could fool yourself into thinking we can just do this on the CPU due to the sparsity of activations. I think the…

My take, for pragmatic reasons rather than how the brain actually works, is that an agent-based architecture is great because some tasks can be solved more effectively by specific algorithms or workflows rather than operating at the low level of neural networks (NN).

Re: Were RNNs all we needed?

#215

Post by refulgentis »

eru wrote: post #190
leereeves wrote: post #143

Earlier quoted context omitted.

> Training your neural net only fiddles with the parameters like a and b. It doesn't do anything about the shape of the function. It doesn't change sine into multiplication etc. It definitely can. The output will always be piecewise linear (with ReLU), but the overall shape can change completely.

Sorry, when I meant 'shape' of the function, I meant the shape of the abstract syntax tree (or something like that). Not the shape of its graph when you draw it.

More directly than my first attempt: you're continuing the error here. The nave's approach of "it's approximating some function" both maps to reality and makes accurate predictions. The more we couple ourselves to "no no no, it's modeling a precise function", the more we end up wrong, both on how it works in theory and in practice.

Re: Were RNNs all we needed?

#216

Post by jszymborski »

YeGoblynQueenne wrote: post #191
theanonymousone wrote: post #158

I remember that, the way I understood it, Transformers solved two major "issues" of RNNs that enabled the later boom: Vanishing gradients limiting the context (and model?) size and difficulty in parallelisation limiting the size of the training data. Do we have solutions for these two problems now?

Vanishing (or exploding) gradients affected all deep architectures, not just RNNs. They were solved by LSTMs first proposed in 1997. See: https://www.semanticscholar.org/paper/Long-Short-Term-Memory... I find it interesting that this knowledge seems to be all but forgotten now. Back in the day, ca. 2014, LSTMs were all the rage, e.g. see: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.i…

> They were solved by LSTMs first proposed in 1997.

I see this stuff everywhere online and it's often taught this way so I don't blame folks for repeating it, but I think it's likely promulgated by folks who don't train LSTMs with long contexts.

LSTMs do add something like a "skip-connection" (before that term was a thing) which helps deal with the catastrophic vanishing gradients you get from e.g. Jordan RNNs right from the jump.

However (!), while this stops us from seeing vanishing gradients after e.g. 10s or 100s of time-steps, when you start seeing multiple 1000s of tokens, the wheels start falling off. I saw this in my own research, training on amino acid sequences of 3,000 length led to a huge amount of instability. It was only after tokenizing the amino acid sequences (which was uncommon at the time) which got us down to ~1500 timesteps on average, did we start seeing stable losses at training. Check-out the ablation at [0].

You can think of ResNets by analogy. ResNets didn't "solve" vanishing gradients, there's a practical limit of the depth of networks, but it did go a long way towards dealing with it.

EDIT: I wanted to add, while I was trying to troubleshoot this for myself, it was super hard to find evidence online of why I was seeing instability. Everything pertaining to "vanishing gradients" and LSTMs were blog posts and pre-prints which just merrily repeated "LSTMs solve the problem of vanishing gradients". That made it hard for me, a junior PhD at the time, to suss out the fact that LSTMs do demonstrably and reliably suffer from vanishing gradients at longer contexts.

[0] https://academic.oup.com/bioinformatics/article/38/16/3958/6...

Re: Were RNNs all we needed?

#217

Post by planetpluta »

seanhunter wrote: post #206
bob1029 wrote: post #196

Earlier quoted context omitted.

Neuromorphic mostly just means "like how the brain works". It encompasses a variety of software & hardware approaches. The most compelling and obvious one to me is hardware purpose-built to simulate spiking neural networks. In the happy case, SNNs are extremely efficient. Basically consuming no energy. You could fool yourself into thinking we can just do this on the CPU due to the sparsity of activations. I think the…

We somehow want a network that is neuromorphic in structure but we don't want it to be like the brain and take 20 years or more to train? Secondly how do we get to claim that a particular thing is neuromorphic when we have such a rudimentary understanding of how a biological brain works or how it generates things like a model of the world, understanding of self etc etc.

Something to consider is that it really could take 20+ years to train like a brain. But once you’ve trained it, you can replicate at ~0 cost, unlike a brain.

Re: Were RNNs all we needed?

#218

Post by kybernetikos »

seanhunter wrote: post #206
bob1029 wrote: post #196

Earlier quoted context omitted.

Neuromorphic mostly just means "like how the brain works". It encompasses a variety of software & hardware approaches. The most compelling and obvious one to me is hardware purpose-built to simulate spiking neural networks. In the happy case, SNNs are extremely efficient. Basically consuming no energy. You could fool yourself into thinking we can just do this on the CPU due to the sparsity of activations. I think the…

We somehow want a network that is neuromorphic in structure but we don't want it to be like the brain and take 20 years or more to train? Secondly how do we get to claim that a particular thing is neuromorphic when we have such a rudimentary understanding of how a biological brain works or how it generates things like a model of the world, understanding of self etc etc.

> we don't want it to be like the brain and take 20 years or more to train?

Estimates put training of gpt4 at something like 2500 gpu years to train, over about 10000 gpus. 20 years would be a big improvement.

Re: Were RNNs all we needed?

#219

Post by x3haloed »

bob1029 wrote: post #22

> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually. > RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction. I would like to draw an analogy to digital signal processing. If you think of the recur…

> I don’t think we get to the exponential scary part of AI without some fundamentally recurrent architecture

I’ve been thinking the same for a while, but I’m starting to wonder if giant context windows are good enough to get us there. I think recurrency is more neuromorphic, and possibly important in the longer run, but maybe not required for SI.

I’m also just a layman with just a surface level understanding of these things, so I may be completely ignorant and wrong.

Re: Were RNNs all we needed?

#220

Post by quantadev »

roboboffin wrote: post #188
quantadev wrote: post #73

Earlier quoted context omitted.

Every LLM expert on the planet agrees LLMs are doing "reasoning". No one says they have feelings or qualia, but we all know there's definitely genuinely artificial reasoning happening. What LLMs have shown both Neuroscience and Computer Science is that reasoning is a mechanical process (or can be simulated by mechanical processes) and is not purely associated only with consciousness.

I'm not sure that's true at all. There are several well known researchers that say LLMs are in fact not doing reasoning.

Those are all the people that have not yet decoupled "reasoning" from "consciousness" in their own way of thinking. It's admittedly hyperbolic to say "everyone". I love hyperbole on HN. :)
Post reply on HN