Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Python code is a neural network

blog.gabornyeki.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: My Python code is a neural network

#61

Post by mr_toad »

scotchmi_st wrote: post #14

This is an interesting article if you read it like a howto for constructing a neural network for performing a practical task. But if you take it at face-value, and follow a similar method the next time you need to parse some input, then, well, I don't know what to say really. The author takes a hard problem (parsing arbitrary input for loosely-defined patterns), and correctly argues that this is likely to produce har…

The spaghetti code approach is basically an expert system. An old school algorithmic AI. Outside constrained domains these systems never really performed very well. Reality is just too messy.

Having a system where you can see why it works the way it does is all very well, but not if it keeps giving the wrong answers. In real world use getting the right answer is often more important than knowing how you got that answer.

Re: My Python code is a neural network

#62

Post by zelphirkalt »

mr_toad wrote: post #61
scotchmi_st wrote: post #14

This is an interesting article if you read it like a howto for constructing a neural network for performing a practical task. But if you take it at face-value, and follow a similar method the next time you need to parse some input, then, well, I don't know what to say really. The author takes a hard problem (parsing arbitrary input for loosely-defined patterns), and correctly argues that this is likely to produce har…

The spaghetti code approach is basically an expert system. An old school algorithmic AI. Outside constrained domains these systems never really performed very well. Reality is just too messy. Having a system where you can see why it works the way it does is all very well, but not if it keeps giving the wrong answers. In real world use getting the right answer is often more important than knowing how you got that answ…

You can make an expert system extensible though. You can make it definitely recognize some pattern and when there is a complaint about it not recognizing another pattern, you can add it. Hopefully you wrote the code in a way that easily allows adding new patterns, of course.

Re: My Python code is a neural network

#63

Post by jb1991 »

someoneontenet wrote: post #35
derangedHorse wrote: post #29

Earlier quoted context omitted.

What other algorithms can do this and which situations would they be more useful than neural networks?

Newtons Method approximates square roots. Its useful if you want to approximate something like that without pulling in the computational power required of NN.

By definition, that’s not a “universal“ function approximator.

Re: My Python code is a neural network

#66

Post by camjw »

dontlikeyoueith wrote: post #58
derangedHorse wrote: post #29

Earlier quoted context omitted.

What other algorithms can do this and which situations would they be more useful than neural networks?

The Taylor Series dates to 1715. Fourier Series dates to the 1820s. Both are universal function approximators and both can be learned via gradient descent. For the case where the function you want to learn actually is polynomial or periodic (respectively), these are better than neural networks.

For your interest, Taylor Series are not universal function approximators - the Taylor Series around 0 for

f(x) = e^(-1/x^2) if x != 0 else 0

is identically zero (all partial derivatives are 0 at 0) but the function is clearly not identically zero. So the radius of convergence for this Taylor series is infinite but it only equals the approximated function at one point.

I'm sure there are some conditions you can put on f to make the Taylor Series a UFA but it's been quite a while since I did any real analysis so I have forgotten!

Doesn't detract from the overall point though that there are UFAs that are not neural nets. I should say that I don't know what the precise definition of a UFA really is, but I assume you have to have more than equality at one point.

Re: My Python code is a neural network

#67

Post by Horffupolde »

camjw wrote: post #66
dontlikeyoueith wrote: post #58

Earlier quoted context omitted.

The Taylor Series dates to 1715. Fourier Series dates to the 1820s. Both are universal function approximators and both can be learned via gradient descent. For the case where the function you want to learn actually is polynomial or periodic (respectively), these are better than neural networks.

For your interest, Taylor Series are not universal function approximators - the Taylor Series around 0 for f(x) = e^(-1/x^2) if x != 0 else 0 is identically zero (all partial derivatives are 0 at 0) but the function is clearly not identically zero. So the radius of convergence for this Taylor series is infinite but it only equals the approximated function at one point. I'm sure there are some conditions you can put o…

Taylor series work on differentiable intervals. You specifically chose a function and interval where this is not true. Of course it will not be a good approximation.

Re: My Python code is a neural network

#68

Post by awwaiid »

OK so first compile python to a NN. But next let's twist or overlay that onto a Transformer-based NN. Then we can have a Transformer Virtual Machine (TVM) execute arbitrary programs.

Use some of that transfer-learning (adding weights on top of each other) and an LLM can be "born" with an algorithm deeply encoded.

Re: My Python code is a neural network

#69

Post by dekhn »

jszymborski wrote: post #57
sailingparrot wrote: post #18

Earlier quoted context omitted.

Not if you want to be a PhD/Researcher in ML, yes otherwise. Source: Working on ML/LLMs as a research engineer for the past 7 years, including for one of the FAANG's research lab, always wanted to take time to learn about RNN but never did and never needed to.

None of the students who have taken the classes I TA pass w/I learning about RNNs.

Is that true also of LSTMs?

Re: My Python code is a neural network

#70

Post by jpe90 »

I recently wrote a blog post exploring the idea of interfacing with local LLMs for ambiguous tasks like this. Doesn't that make more sense than coding the neural network yourself? Using something like llama.cpp and evaluating whether a small model solves your problem out of the box, and fine-tuning if not, then programmatically interfacing with llama.cpp via a wrapper of your choice seems more pragmatic to me.
Post reply on HN