Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why train when you can optimize?

justinmeiners.github.io

Post reply on HN
Print view View original

11–20 of 70 posts

Re: Why train when you can optimize?

#11

Post by yobbo »

whatshisface wrote: post #6
steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

If we're allowed to do that , why can't my activation function be sin(x)?

It can, but sin(x) has infinite number of extremes, and the gradients will vanish at those points. Activations will get stuck at 1 and -1 (x=π/2, 3π/2, ...). They set x+(1/a)*sin²(x) to be monotonic, which fixes this.

Or you need to optimize without using gradients.

Re: Why train when you can optimize?

#12

Post by zarzavat »

blackcat201 wrote: post #9
whatshisface wrote: post #6

Earlier quoted context omitted.

If we're allowed to do that , why can't my activation function be sin(x)?

You can use sin as activation function, but that would require careful initialization to avoid gradient explosion as you would ended up with a lot of points where gradient is simply zero. You can refer to Implicit Neural Representations with Periodic Activation Functions for more details.

It works perfectly if you don’t have any parameters.

Re: Why train when you can optimize?

#13

Post by telchar »

SemanticStrengh wrote: post #10
steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the s…

And yet neural networks can solve symbolic integral and derivative problems and differential equations better than other computer algebra programs. Sure one network might fail to compute sin(x) numerically, but another could easily tell you its derivative is cos(x). Turns out they are pretty flexible. Do they need to do everything?

Re: Why train when you can optimize?

#14

Post by viraptor »

SemanticStrengh wrote: post #10
steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the s…

> And then your NN can't represent anything else than periodic functions

Significant parts of x+sin are close to linear. You don't need to use this activation as the result layer either. Why would we lose anything?

> If we had to build separate programs for each product requirements variations

We pretty much do? We use both extremely generic frameworks both in technical sense (.net) and organisational (sap). But we also have software written to specific requirements where needed (there's millions of very specific ways to invoice someone, companies get invoicing platforms written just for them from scratch). There's space for both approaches.

Re: Why train when you can optimize?

#15

Post by PartiallyTyped »

whatshisface wrote: post #6
steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

If we're allowed to do that , why can't my activation function be sin(x)?

You can actually do that.

https://www.vincentsitzmann.com/siren/

Re: Why train when you can optimize?

#16

Post by PartiallyTyped »

SemanticStrengh wrote: post #10
steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the s…

You are wrong.

https://www.vincentsitzmann.com/siren

Re: Why train when you can optimize?

#17

Post by SemanticStrengh »

PartiallyTyped wrote: post #16
SemanticStrengh wrote: post #10

Earlier quoted context omitted.

And then your NN can't represent anything else than periodic functions.. If we had to build separate programs for each product requirements variations.. programming would not be viable. More generally neural networks can't even imitate a dumb calculator without throwing absurd errors despite the rules of calculus being trivial and well defined. And matching a calculator is a task order of magnitudes easier than the s…

You are wrong. https://www.vincentsitzmann.com/siren

? I did not say that NNs can't represent periodic functions. Show me a NN calculator with >99% accuracy if you wanna refute me.

Re: Why train when you can optimize?

#18

Post by zarzavat »

steppi wrote: post #4

Neural networks can approximate any function, but that doesn’t mean they do so efficiently. Depending on the function, they can require incredible amounts of neurons and training. At their worst, they devolve into a lookup table. It’s not hard to find these examples either. Just try training a neural network to compute sin(x)! This is possible! One of the cool things about neural networks is that you can try to encod…

It’s hard to walk if you can’t feel your legs.

The problem is improperly evaluating the neural network as a function of time, instead of evaluating the network as a function of previous state.

When we humans approximate functions (let’s say you’re drawing it on a piece of paper, or waving your arm around) we do not simply look at a clock and feed forward that information directly into our motor neurons. Rather, we have sensory neurons that feed in the current state of the function we are approximating as an input, then approximating the next output of a periodic function becomes trivial.

It’s very easy to train a neural network to look at a piece of a sine wave and predict what the next value should be - the fact that the function is periodic actually helps you.

Re: Why train when you can optimize?

#20

Post by sicp-enjoyer »

PartiallyTyped wrote: post #15
whatshisface wrote: post #6

Earlier quoted context omitted.

If we're allowed to do that , why can't my activation function be sin(x)?

You can actually do that. https://www.vincentsitzmann.com/siren/

I'm surprised to see almost no discussion of fourier series in that paper, considering fourier series is all about representing signals as linear combinations of sinusoidal functions.
Post reply on HN