I'd like to see a cost vs precision/recall comparison of using a RNN vs an LLM (local or API) for a problem like this.
Hah... Ask, and HN (sorta) delivers. Not RNN, but self-finetuned LLM cost vs performance compared to GPT-4.
My Python code is a neural network
51–60 of 75 posts
Re: My Python code is a neural network
#52> To model the state, we need to add three hidden layers to the network Why 3? And why use "h" for layer names?
`h` is for "hidden" layers.
Re: My Python code is a neural network
#53[flagged]
Re: My Python code is a neural network
#54Earlier quoted context omitted.
FYI, there are actually many algorithms going back longer than the neural network algorithm that have been proven to be a universal function approximator. Neural networks are certainly not the only and not the first to do so. There are quite a few that are actually much more appropriate for many cases than a neural network.
What other algorithms can do this and which situations would they be more useful than neural networks?
Re: My Python code is a neural network
#55Earlier quoted context omitted.
Not if you want to be a PhD/Researcher in ML, yes otherwise. Source: Working on ML/LLMs as a research engineer for the past 7 years, including for one of the FAANG's research lab, always wanted to take time to learn about RNN but never did and never needed to.
Oh, I'm sure plenty of recent PhDs don't know about RNNs. They've been dropped like a hot potato in the last 4-5 years.
I think to do pure research it’s definitely worth knowing about the big ideas of the past, why we moved on from them, what we learned etc.
Re: My Python code is a neural network
#56That’s pretty lightwork for a snake in general. Send me a direct message if you come up with something better.
Re: My Python code is a neural network
#57Are RNNs completely subsumed by transformers? IE, can I forget about learning anything about how to work with RNNs, and instead focus on transformers?
Not if you want to be a PhD/Researcher in ML, yes otherwise. Source: Working on ML/LLMs as a research engineer for the past 7 years, including for one of the FAANG's research lab, always wanted to take time to learn about RNN but never did and never needed to.
None of the students who have taken the classes I TA pass w/I learning about RNNs.
Re: My Python code is a neural network
#58Earlier quoted context omitted.
FYI, there are actually many algorithms going back longer than the neural network algorithm that have been proven to be a universal function approximator. Neural networks are certainly not the only and not the first to do so. There are quite a few that are actually much more appropriate for many cases than a neural network.
What other algorithms can do this and which situations would they be more useful than neural networks?
The Taylor Series dates to 1715. Fourier Series dates to the 1820s.
Both are universal function approximators and both can be learned via gradient descent.
For the case where the function you want to learn actually is polynomial or periodic (respectively), these are better than neural networks.
Re: My Python code is a neural network
#59I would love to see some work on duality i.e. code to neural network back and forth. Reason being - I can't debug a neural network but if it can be linearized into a if-else case with help of the token information => I can validate what it's doing -> fix it and then move it back to it's compressed neural representation.
Just another thought experiment -> sometimes I imagine neural networks as a zip of the training data where compression algorithm is backpropagation. Just like we have programs which let us see what files inside the zip are -> I imagine there can be programs which will let us select certain inference path of the neural net and then see what data affected that => then we edit that data to fix our issues or add more data there => and we have live neural network debugging and reprogramming in the same way we edit compressed zips
Re: My Python code is a neural network
#60Next time do one with Bayesian networks or another Probabilistic graphical model.