Why train when you can optimize? • Hacker News Board

Re: Why train when you can optimize?

#21

Post by PartiallyTyped » Sat, Apr 30, 2022, 6:09 AM UTC

SemanticStrengh wrote: post #17

PartiallyTyped wrote: post #16

Earlier quoted context omitted.

You are wrong. https://www.vincentsitzmann.com/siren

? I did not say that NNs can't represent periodic functions. Show me a NN calculator with >99% accuracy if you wanna refute me.

> and then your NN can't represent anything else than periodic functions..

This is a quote from the parent comment where you state that NNs with periodic activations can't represent non periodic functions.

Siren (what I linked) uses periodic activations and is able to represent non periodic functions.

Re: Why train when you can optimize?

#22

Post by diffeomorphism » Sat, Apr 30, 2022, 6:18 AM UTC

taeric wrote: post #5

Isn't training literally an optimization of a loss function over training data?

Technically yes, but practically no. This is discussed in depth in section 11.

Re: Why train when you can optimize?

#23

Post by PartiallyTyped » Sat, Apr 30, 2022, 6:22 AM UTC

sicp-enjoyer wrote: post #20

PartiallyTyped wrote: post #15

Earlier quoted context omitted.

You can actually do that. https://www.vincentsitzmann.com/siren/

I'm surprised to see almost no discussion of fourier series in that paper, considering fourier series is all about representing signals as linear combinations of sinusoidal functions.

You may be interested in [1] where they go to a great extend to show that the convolution operation that we consider in DL is the dual of fourier series [2].

[1] https://geometricdeeplearning.com

[2] https://arxiv.org/pdf/2104.13478.pdf page 27 (23 if you count book pages).

Re: Why train when you can optimize?

#24

Post by SemanticStrengh » Sat, Apr 30, 2022, 6:33 AM UTC

PartiallyTyped wrote: post #21

SemanticStrengh wrote: post #17

Earlier quoted context omitted.

? I did not say that NNs can't represent periodic functions. Show me a NN calculator with >99% accuracy if you wanna refute me.

> and then your NN can't represent anything else than periodic functions.. This is a quote from the parent comment where you state that NNs with periodic activations can't represent non periodic functions. Siren (what I linked) uses periodic activations and is able to represent non periodic functions.

Oh OK, thanks yes that's what I was looking for! However if this NN can represent non Periodic functions, what cost does the Periodic function support incur on non Periodic function accuracy, if any?

Re: Why train when you can optimize?

#25

Post by omrjml » Sat, Apr 30, 2022, 6:41 AM UTC

zarzavat wrote: post #18

steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

It’s hard to walk if you can’t feel your legs. The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of previous state. When we humans approximate functions (let’s say you’re drawing it on a piece of paper, or waving your arm around) we do not simply look at a clock and feed forward that information directly into our motor neurons. Rather, we ha…

Periodic function like sin(x) are not a dynamical system so its previous state does not determine the current state. So it should be approximated in that way.

Re: Why train when you can optimize?

#26

Post by doubleunplussed » Sat, Apr 30, 2022, 6:48 AM UTC

omrjml wrote: post #25

zarzavat wrote: post #18

Earlier quoted context omitted.

It’s hard to walk if you can’t feel your legs. The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of previous state. When we humans approximate functions (let’s say you’re drawing it on a piece of paper, or waving your arm around) we do not simply look at a clock and feed forward that information directly into our motor neurons. Rather, we ha…

Periodic function like sin(x) are not a dynamical system so its previous state does not determine the current state. So it should be approximated in that way.

Periodic functions like sin(x) are the solutions to differential equations like dy/dx = -y that describe for example, oscillations of springs, to name but one of an extremely large number of dynamical systems that behave this way.

Re: Why train when you can optimize?

#27

Post by omrjml » Sat, Apr 30, 2022, 6:49 AM UTC

omrjml wrote: post #25

zarzavat wrote: post #18

Earlier quoted context omitted.

It’s hard to walk if you can’t feel your legs. The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of previous state. When we humans approximate functions (let’s say you’re drawing it on a piece of paper, or waving your arm around) we do not simply look at a clock and feed forward that information directly into our motor neurons. Rather, we ha…

Periodic function like sin(x) are not a dynamical system so its previous state does not determine the current state. So it should be approximated in that way.

Edit: should not be approximated

Re: Why train when you can optimize?

#28

Post by ABeeSea » Sat, Apr 30, 2022, 6:50 AM UTC

telchar wrote: post #13

SemanticStrengh wrote: post #10

Earlier quoted context omitted.

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the s…

And yet neural networks can solve symbolic integral and derivative problems and differential equations better than other computer algebra programs. Sure one network might fail to compute sin(x) numerically, but another could easily tell you its derivative is cos(x). Turns out they are pretty flexible. Do they need to do everything?

Source? I would be very surprised if there was a neural symbolic PDE solver better than what’s in wolfram mathematica.

Re: Why train when you can optimize?

#29

Post by mhh__ » Sat, Apr 30, 2022, 6:53 AM UTC

telchar wrote: post #13

SemanticStrengh wrote: post #10

Earlier quoted context omitted.

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the s…

And yet neural networks can solve symbolic integral and derivative problems and differential equations better than other computer algebra programs. Sure one network might fail to compute sin(x) numerically, but another could easily tell you its derivative is cos(x). Turns out they are pretty flexible. Do they need to do everything?

At the cost of how many parameters?

Re: Why train when you can optimize?

#30

Post by taylorius » Sat, Apr 30, 2022, 6:54 AM UTC

A small shout-out to Differential Evolution, which is my go-to (derivative free) optimization algorithm. Think of a Genetic Algorithm, but the crossover operator is linear interpolation (DE's natural domain is real-valued vectors). It's simple, and in my experience, it works pretty well.