Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Were RNNs all we needed?

arxiv.org

Post reply on HN
Print view View original

101–110 of 269 posts

Re: Were RNNs all we needed?

#101

Post by drodgers »

xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

> The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape

I have almost the opposite take. We've had a lot of datasets for ages, but all the progress in the last decade has come from advances how curves are architected and fit to the dataset (including applying more computing power).

Maybe there's some theoretical sense in which older models could have solved newer problems just as well if only we applied 1000000x the computing power, so the new models are 'just' an optimisation, but that's like dismissing the importance of complexity analysis in algorithm design, and thus insisting that bogosort and quicksort are equivalent.

When you start layering in normalisation techniques to minimise overfitting, and especially once you start thinking about more agentic architectures (eg. Deep Q Learning, some of the search space design going into OpenAI's o1), then I don't think the just-an-optimisation perspective can hold much water at all - more computing power simply couldn't solve those problems with older architectures.

Re: Were RNNs all we needed?

#103

Post by quantadev »

jcparkyn wrote: post #99
quantadev wrote: post #96

Earlier quoted context omitted.

> The non linearities are not there primarily to keep the outputs in a given range Precisely what the `Activation Function` does is to squash an output into a range (normally below one, like tanh). That's the only non-linearity I'm aware of. What other non-linearities are there? All the training does is adjust linear weights tho, like I said. All the training is doing is adjusting the slopes of lines.

> squash an output into a range This isn't the primary purpose of the activation function, and in fact it's not even necessary. For example see ReLU (probably the most common activation function), leaky ReLU, or for a sillier example: https://youtu.be/Ae9EKCyI1xU?si=KgjhMrOsFEVo2yCe

You can change the subject by bringing up as many different NN architectures, Activation Functions, etc. as you want. I'm telling you the basic NN Perceptron design (what everyone means when they refer to Perceptrons in general), has something like a `tanh` and not only is it's PRIMARY function to squash a number, that's it's ONLY function.

Re: Were RNNs all we needed?

#104

Post by DAGdug »

exe34 wrote: post #36
DAGdug wrote: post #21

Earlier quoted context omitted.

“ I've overturned the field leader several times in science” Either that makes you a field leader yourself, or you did it for trivial things, or you’re BSing. Which one is it?

there's a big space between leader and trivial. it's entirely possible to point out the top leader in your field is wrong on ten things over a career, without becoming the top leader yourself.

On speculative things or trivial things, sure! On substantive matters (recall: the choice of words is “overturned”), in empirical realms or theory (physics, CS) or math, it’s rather doubtful. Anonymous, self-declared geniuses aren’t to be taken at face value.

Re: Were RNNs all we needed?

#105

Post by bee_rider »

TaurenHunter wrote: post #51
ants_everywhere wrote: post #46

Earlier quoted context omitted.

> is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning) (Somewhat) fun and (somewhat) related fact: there's a whole cottage industry of "is all you need" papers https://arxiv.org/search/?query=%22is+all+you+need%22&search...

Reminds me of the "Considered Harmful" articles: https://meyerweb.com/eric/comment/chech.html

Quick, somebody write “All you need Considered Harmful” and “Considered Harmful all you need.”

Which seems closer to true?

Re: Were RNNs all we needed?

#106

Post by slashdave »

trott wrote: post #98
slashdave wrote: post #87

Earlier quoted context omitted.

> the amount of information the model retains about it is bounded by whatever is in its hidden state This is no different than a transformer, which, after all, is bound by a finite state, just organized in a different manner.

> This is no different than a transformer, which, after all, is bound by a finite state, just organized in a different manner. It's not just a matter of organizing things differently. Suppose your network dimension and sequence length are both X. Then your memory usage (per layer) will be O(X^2), while your training update cost will be O(X^3). That's for both Transformers and RNNs. However, at the end of the sequence…

Simplistic thinking. An RNN hidden parameter space of high dimension provides plenty of room for linear projections of token histories. I think people just do not realize just how huge R^N can be.

Re: Were RNNs all we needed?

#107

Post by vandahm »

I made a RNN for a college project because I was interested in obsolete historical technology and I thought I needed to seize the opportunity while it lasted, because once I was out of school, I'd never hear about neural networks ever again.

Mine worked, but it was very simple and dog slow, running on my old laptop. Nothing was ever going to run fast on that thing, but I remember my RNN being substantially slower than a feed-forward network would have been.

I was so confident that this was dead technology -- an academic curiosity from the 1980s and 1990s. It was bizarre to see how quickly that changed.

Re: Were RNNs all we needed?

#108

Post by mr_toad »

ctur wrote: post #95
xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

Architecture matters because while deep learning can conceivably fit a curve with a single, huge layer (in theory... Universal approximation theorem), the amount of compute and data needed to get there is prohibitive. Having a good architecture means the theoretical possibility of deep learning finding the right N dimensional curve becomes a practical reality. Another thing about the architecture is we inherently bia…

> can conceivably fit a curve with a single, huge layer

I think you need a hidden layer. I’ve never seen a universal approximation theorem for a single layer network.

Re: Were RNNs all we needed?

#109

Post by f_devd »

trott wrote: post #98
slashdave wrote: post #87

Earlier quoted context omitted.

> the amount of information the model retains about it is bounded by whatever is in its hidden state This is no different than a transformer, which, after all, is bound by a finite state, just organized in a different manner.

> This is no different than a transformer, which, after all, is bound by a finite state, just organized in a different manner. It's not just a matter of organizing things differently. Suppose your network dimension and sequence length are both X. Then your memory usage (per layer) will be O(X^2), while your training update cost will be O(X^3). That's for both Transformers and RNNs. However, at the end of the sequence…

Transformers actually have an quantifiable state size (see https://hazyresearch.stanford.edu/static/posts/2024-06-22-ac...) although it's anywhere between 200k and 2M floats (for 360M and 1.33B respectively iinm). So a sufficiently sized RNN could have the same state capacity as a transformer.

(this is from the Based paper: https://arxiv.org/pdf/2402.18668)

Re: Were RNNs all we needed?

#110

Post by _giorgio_ »

xnx wrote: post #11

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't f…

Chollet is just a philosopher. He also thinks that keras and tensorflow are important, when nobody uses those. And he punished false days about their usage.
Post reply on HN