Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

21–30 of 269 posts

Re: Were RNNs all we needed?

#21

Post by DAGdug »

hotspot_one wrote: post #16
auggierose wrote: post #9

Earlier quoted context omitted.

And this is where science breaks down.

Not really, because 1) Yoshua's reputation would take a hit if this paper were bullshit, so he has extrinsic motivation to make it good 2) Yoshua has enough experience in the field to know what is going on in the field, you don't have to ask if he forgot about a certain architecture or the work of a certain research group which would contradict his findings-- if such work exists and is credible, it is very likely to…

“ I've overturned the field leader several times in science” Either that makes you a field leader yourself, or you did it for trivial things, or you’re BSing. Which one is it?

Re: Were RNNs all we needed?

#22

Post by bob1029 »

> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually.

> RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction.

I would like to draw an analogy to digital signal processing. If you think of the recurrent-style architectures as IIR filters and feedforward-only architectures as FIR filters, you will likely find many parallels.

The most obvious to me being that IIR filters typically require far fewer elements to produce the same response as an equivalent FIR filter. Granted, the FIR filter is often easier to implement/control/measure in practical terms (fixed-point arithmetic hardware == ML architectures that can run on GPUs).

I don't think we get to the exponential scary part of AI without some fundamentally recurrent architecture. I think things like LSTM are kind of an in-between hack in this DSP analogy - You could look at it as FIR with dynamic coefficients. Neuromorphic approaches seem like the best long term bet to me in terms of efficiency.

Re: Were RNNs all we needed?

#23

Post by baanist »

marcosdumay wrote: post #18

R == Recurrent From theory the answer to the question should be "yes", they are Turing complete. The real question is about how to train them, and the paper is about that.

Why aren't AI researchers automating the search for efficient architectures?

Re: Were RNNs all we needed?

#24

Post by auggierose »

hotspot_one wrote: post #16
auggierose wrote: post #9

Earlier quoted context omitted.

And this is where science breaks down.

Not really, because 1) Yoshua's reputation would take a hit if this paper were bullshit, so he has extrinsic motivation to make it good 2) Yoshua has enough experience in the field to know what is going on in the field, you don't have to ask if he forgot about a certain architecture or the work of a certain research group which would contradict his findings-- if such work exists and is credible, it is very likely to…

> It only means we cannot dismiss it out of hand, if we have a different view we would need substantive arguments to defend our view.

You will need to do that anyway, no matter if Yoshua is on the paper, or not. I understand that people have limited bandwidth, and so they need shortcuts, and they need to justify these shortcuts to themselves somehow (of course the justifications are nonsense). Maybe AI will help here.

Re: Were RNNs all we needed?

#25

Post by teruakohatu »

islewis wrote: post #15
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

> "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime." I haven't fully ingested the paper yet, but it looks like it's focused more on compute optimization than the size of the dataset: > ... and (2) are fully parallelizable during training (175x faster for a sequence of length 512 Even if many types of architectures converge to the same l…

> Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale.

This! Not just fastest but with the lowest resources in total.

Fully connected neural networks are universal functions. Technically we don’t need anything but a FNN, but memory requirements and speed would be abysmal far beyond the realm of practicality.

Re: Were RNNs all we needed?

#26

Post by vundercind »

dsamarin wrote: post #17

The name of the paper contrasts with the paper that spawned Transformer architecture, which itself is a reference to the song "All You Need Is Love" by the Beatles. https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

I eagerly await the backlash to suggesting any one thing is all you need, the first shot of which shall surely be titled: “‘All you need’ Considered Harmful”

Re: Were RNNs all we needed?

#28

Post by fsndz »

xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

[flagged]

Re: Were RNNs all we needed?

#29

Post by adamnemecek »

Yes, all machine learning can be interpreted in terms of approximating the partition function.

This is obvious when one considers the connections between Transformers, RNNs, Hopfield networks and the Ising model, a model from statistical mechanics which is solved by calculating the partition function.

This interpretation provides us with some very powerful tools that are commonplace in math and physics but which are not talked about in CS & ML.

I'm working on a startup http://traceoid.ai which takes this exact view. Our approach enables faster training and inference, interpretability and also scalable energy-based models, the Holy Grail of machine learning.

Join the discord https://discord.com/invite/mr9TAhpyBW or follow me on twitter https://twitter.com/adamnemecek1

Post reply on HN