Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Python code is a neural network

blog.gabornyeki.com

Post reply on HN
Print view View original

11–20 of 75 posts

Re: My Python code is a neural network

#11

Post by danans »

alsxnt wrote: post #6

Recurrent neural networks can be used for arbitrary computations, the equivalence to Turing machines has been proven. However, they are utterly impractical for the task. This seems to be a state machine that is somehow learned. The article could benefit from a longer synopsis and "Python" does not appear to be relevant at all. Learning real Python semantics would prove quite difficult due to the nature of the languag…

> Recurrent neural networks can be used for arbitrary computations, the equivalence to Turing machines has been proven. However, they are utterly impractical for the task.

Karpathy's 2015 RNN article [1] demonstrated that RNNs trained character-wise on Shakespeare's works could produce Shakespeare-esque text (albeit without the narrative coherence of LLMs). Given that, why wouldn't they be able to handle natural language as formulaic as code review comments?

In that case inference was run with randomized inputs in order to generate random "Shakespeare", but the structure of the language and style was still learned by the RNN. Perhaps it could be used for classification also.

1. https://karpathy.github.io/2015/05/21/rnn-effectiveness/

Re: My Python code is a neural network

#12

Post by pakl »

There exists the Universal (Function) Approximation Theorem for neural networks — which states that they can represent/encode any function to a desired level of accuracy[0].

However there does not exist a theorem stating that those approximations can be learned (or how).

[0] https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

Re: My Python code is a neural network

#13

Post by arketyp »

pakl wrote: post #12

There exists the Universal (Function) Approximation Theorem for neural networks — which states that they can represent/encode any function to a desired level of accuracy[0]. However there does not exist a theorem stating that those approximations can be learned (or how). [0] https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

Makes you wonder what is meant by learning...

Re: My Python code is a neural network

#14

Post by scotchmi_st »

This is an interesting article if you read it like a howto for constructing a neural network for performing a practical task. But if you take it at face-value, and follow a similar method the next time you need to parse some input, then, well, I don't know what to say really.

The author takes a hard problem (parsing arbitrary input for loosely-defined patterns), and correctly argues that this is likely to produce hard-to-read 'spaghetti' code.

They then suggest replacing that with code that is so hard to read that there is still active research into how it works, (i.e a neural net).

Don't over-index something that's inscrutable versus something that you can understand but is 'ugly'. Sometimes, _maybe_, a ML model is what you want for a task. But a lot of the time, something that you can read and see why it's doing what it's doing, even if that takes some effort, is better than something that's impossible.

Re: My Python code is a neural network

#15

Post by dekhn »

arketyp wrote: post #13
pakl wrote: post #12

There exists the Universal (Function) Approximation Theorem for neural networks — which states that they can represent/encode any function to a desired level of accuracy[0]. However there does not exist a theorem stating that those approximations can be learned (or how). [0] https://en.m.wikipedia.org/wiki/Universal_approximation_theo...

Makes you wonder what is meant by learning...

Learning is using observations to create/update a model that makes predictions which are more accurate than chance. At some point the model ends up having generalizability beyond the domain.

Re: My Python code is a neural network

#16

Post by jlturner »

If this interests you, it’s worth taking a look at Genetic Programming. I find it to be a simpler approach at the same problem, no math required. It simply recombines programs by their AST, and given some heuristic, optimizes the program for it. The magic is in your heuristic function, where you can choose what you want to optimize for (ie. Speed, program length, minimize complex constructs or function calls, network efficiency, some combination therein, etc).

https://youtu.be/tTMpKrKkYXo

Re: My Python code is a neural network

#18

Post by sailingparrot »

dekhn wrote: post #17

Are RNNs completely subsumed by transformers? IE, can I forget about learning anything about how to work with RNNs, and instead focus on transformers?

Not if you want to be a PhD/Researcher in ML, yes otherwise.

Source: Working on ML/LLMs as a research engineer for the past 7 years, including for one of the FAANG's research lab, always wanted to take time to learn about RNN but never did and never needed to.

Re: My Python code is a neural network

#19

Post by meindnoch »

scotchmi_st wrote: post #14

This is an interesting article if you read it like a howto for constructing a neural network for performing a practical task. But if you take it at face-value, and follow a similar method the next time you need to parse some input, then, well, I don't know what to say really. The author takes a hard problem (parsing arbitrary input for loosely-defined patterns), and correctly argues that this is likely to produce har…

"Let's tuck all that unsightly spaghetti code behind this neat 1000x1000 matrix of floats!"

Re: My Python code is a neural network

#20

Post by vidarh »

danans wrote: post #11
alsxnt wrote: post #6

Recurrent neural networks can be used for arbitrary computations, the equivalence to Turing machines has been proven. However, they are utterly impractical for the task. This seems to be a state machine that is somehow learned. The article could benefit from a longer synopsis and "Python" does not appear to be relevant at all. Learning real Python semantics would prove quite difficult due to the nature of the languag…

> Recurrent neural networks can be used for arbitrary computations, the equivalence to Turing machines has been proven. However, they are utterly impractical for the task. Karpathy's 2015 RNN article [1] demonstrated that RNNs trained character-wise on Shakespeare's works could produce Shakespeare-esque text (albeit without the narrative coherence of LLMs). Given that, why wouldn't they be able to handle natural lang…

For RNN abilities, RWKV is worth a look[1]

It's billed as "an RNN with GPT-level LLM performance".

[1] https://www.rwkv.com/

Post reply on HN