Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Python code is a neural network

blog.gabornyeki.com

Post reply on HN
Print view View original

31–40 of 75 posts

Re: My Python code is a neural network

#31

Post by derangedHorse »

sailingparrot wrote: post #18
dekhn wrote: post #17

Are RNNs completely subsumed by transformers? IE, can I forget about learning anything about how to work with RNNs, and instead focus on transformers?

Not if you want to be a PhD/Researcher in ML, yes otherwise. Source: Working on ML/LLMs as a research engineer for the past 7 years, including for one of the FAANG's research lab, always wanted to take time to learn about RNN but never did and never needed to.

I haven’t read it in a while but I remember this post giving a good rundown of rnns

https://dennybritz.com/posts/wildml/recurrent-neural-network...

Re: My Python code is a neural network

#33

Post by gnyeki »

derangedHorse wrote: post #29
jb1991 wrote: post #22

Earlier quoted context omitted.

FYI, there are actually many algorithms going back longer than the neural network algorithm that have been proven to be a universal function approximator. Neural networks are certainly not the only and not the first to do so. There are quite a few that are actually much more appropriate for many cases than a neural network.

What other algorithms can do this and which situations would they be more useful than neural networks?

This area is covered by non-parametric statistics more generally. There are many other methods to non-parametrically estimate functions (that satisfy some regularity conditions). Tree-based methods are one family of such methods, and the consensus still seems to be that they perform better than neural networks on tabular data. For example:

https://arxiv.org/abs/2106.03253

Re: My Python code is a neural network

#34

Post by gnyeki »

Fripplebubby wrote: post #8

Love this post! Gets into the details of what it _really_ means to take some function and turn it into an RNN, and comparing that to the "batteries included" RNNs included in PyTorch, as a learning experience. Question: > To model the state, we need to add three hidden layers to the network. How did you determine that it would be three hidden layers? Is it a consequence of the particular rule you were implementing, o…

I'm glad you found it valuable! Both are good questions and I haven't gone far enough mapping the code to Elman's architecture to know the answer to the second.

For your first question, using three hidden layers makes it a little clearer what the network does. Each layer performs one step of the calculation. The first layer collects what is known from the current token and what we knew after the calculation for the previous token. The second layer decides whether the current token looks like program code, by checking if it satisfies the decision rule. The third layer compares the decision with what we decided for previous tokens.

I think that this could be compressed into a single hidden layer, too. A ReLU should be good enough at capturing non-linearities so this should work.

Re: My Python code is a neural network

#35

Post by someoneontenet »

derangedHorse wrote: post #29
jb1991 wrote: post #22

Earlier quoted context omitted.

FYI, there are actually many algorithms going back longer than the neural network algorithm that have been proven to be a universal function approximator. Neural networks are certainly not the only and not the first to do so. There are quite a few that are actually much more appropriate for many cases than a neural network.

What other algorithms can do this and which situations would they be more useful than neural networks?

Newtons Method approximates square roots. Its useful if you want to approximate something like that without pulling in the computational power required of NN.

Re: My Python code is a neural network

#36

Post by toxik »

dekhn wrote: post #17

Are RNNs completely subsumed by transformers? IE, can I forget about learning anything about how to work with RNNs, and instead focus on transformers?

Transformers have finite context, RNNs don’t. In practice the RNN gradient signal is limited by back propagation through time, it decays. This is in fact the whole selling point of transformers; association is not harder or easier in near/short distance. But in theory a RNN can remember infinitely far away.

Re: My Python code is a neural network

#37

Post by Fripplebubby »

dekhn wrote: post #17

Are RNNs completely subsumed by transformers? IE, can I forget about learning anything about how to work with RNNs, and instead focus on transformers?

To further problematize this question (which I don't feel like I can actually answer), consider this paper: "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention" - https://arxiv.org/pdf/2006.16236

What this shows is that actually a specific narrow definition of transformer (a transformer with "causal masking" - see paper) is equivalent to an RNN, and vice versa.

Similarly Mamba (https://arxiv.org/abs/2312.00752), the other hot architecture at the moment, has an equivalent unit to a gated RNN. For performance reasons, I believe they use an equivalent CNN during training and an RNN during inference!

Re: My Python code is a neural network

#38

Post by Fripplebubby »

gnyeki wrote: post #34
Fripplebubby wrote: post #8

Love this post! Gets into the details of what it _really_ means to take some function and turn it into an RNN, and comparing that to the "batteries included" RNNs included in PyTorch, as a learning experience. Question: > To model the state, we need to add three hidden layers to the network. How did you determine that it would be three hidden layers? Is it a consequence of the particular rule you were implementing, o…

I'm glad you found it valuable! Both are good questions and I haven't gone far enough mapping the code to Elman's architecture to know the answer to the second. For your first question, using three hidden layers makes it a little clearer what the network does. Each layer performs one step of the calculation. The first layer collects what is known from the current token and what we knew after the calculation for the p…

Ah, that makes sense. So, we consider two hidden layers more as "memory" or "buffers", and actually the rule is implemented in just one layer, at least for a single token.

Re: My Python code is a neural network

#39

Post by visarga »

pakl wrote: post #12

There exists the Universal (Function) Approximation Theorem for neural networks — which states that they can represent/encode any function to a desired level of accuracy[0]. However there does not exist a theorem stating that those approximations can be learned (or how). [0] https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

They can model only continuous functions, more specifically any continuous function on compact subsets of ℝⁿ. They can approximate functions to an arbitrary level of accuracy, given sufficient neurons

Re: My Python code is a neural network

#40

Post by visarga »

Fripplebubby wrote: post #37
dekhn wrote: post #17

Are RNNs completely subsumed by transformers? IE, can I forget about learning anything about how to work with RNNs, and instead focus on transformers?

To further problematize this question (which I don't feel like I can actually answer), consider this paper: "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention" - https://arxiv.org/pdf/2006.16236 What this shows is that actually a specific narrow definition of transformer (a transformer with "causal masking" - see paper) is equivalent to an RNN, and vice versa. Similarly Mamba ( https://arx…

There still are important distinctions. RNNs have constant memory while transformers expand their memory with each new token. They are related, but one could in theory process an unbounded sequence while the other cannot because of growing memory usage.
Post reply on HN