11–20 of 20 posts
I found that the statement about the cross entropy not true. When y==a the function is non-monotonic with 0 at the extremes but not at the middle. So the "proof" shown is confusing to me.
Which statement about cross entropy is confusing?
The statement "if the neuron's actual output is close to the desired output, i.e., y=y(x) for all training inputs x, then the cross-entropy will be close to zero"
is not true. The function peaks in the middle (~ 0.7)
Thanks! -Kaushik
Earlier quoted context omitted.
Which statement about cross entropy is confusing?
Hi! The statement "if the neuron's actual output is close to the desired output, i.e., y=y(x) for all training inputs x, then the cross-entropy will be close to zero" is not true. The function peaks in the middle (~ 0.7) Thanks! -Kaushik
The essential point is that we're considering classification problems, for which the output is intended to be 0 or 1. I address the more general case of regression problems (where y may take any value) in a later exercise.
Hope that helps!
Neural Networks class on coursera covered a lot of the same topics with both heavy math theory crafting and hefty amount of practical application. https://www.coursera.org/course/neuralnets
It's a pity the lectures are not accessible any longer. How i wish i could find them somewhere!
Neural Networks class on coursera covered a lot of the same topics with both heavy math theory crafting and hefty amount of practical application. https://www.coursera.org/course/neuralnets
It's a pity the lectures are not accessible any longer. How i wish i could find them somewhere!
And then you can download them with coursistant or coursera-dl and have them wherever you go...
Earlier quoted context omitted.
Hi! The statement "if the neuron's actual output is close to the desired output, i.e., y=y(x) for all training inputs x, then the cross-entropy will be close to zero" is not true. The function peaks in the middle (~ 0.7) Thanks! -Kaushik
This is addressed in the marginal note attached to the sentence you quoted. The essential point is that we're considering classification problems, for which the output is intended to be 0 or 1. I address the more general case of regression problems (where y may take any value) in a later exercise. Hope that helps!
>You have to realize that our theoretical tools are very weak. Sometimes, we have good mathematical intuitions for why a particular technique should work. Sometimes our intuition ends up being wrong [...] The questions become: how well does my method work on this particular problem, and how large is the set of problems on which it works well. I'm not very familiar with this field. Has anyone made any progress on form…
>You have to realize that our theoretical tools are very weak. Sometimes, we have good mathematical intuitions for why a particular technique should work. Sometimes our intuition ends up being wrong [...] The questions become: how well does my method work on this particular problem, and how large is the set of problems on which it works well. I'm not very familiar with this field. Has anyone made any progress on form…
[deleted]
I beg to differ sir: http://nlp.stanford.edu/pubs/SocherGanjooManningNg_NIPS2013....
This paper demonstrates something called "0 shot learning" where you can actually infer the correct label of an unseen image based on similarity among representations learned in a separate NLP task.
For instance, it can label an image "tiger" even if it has not seen tigers but has only learned about the word (and inferred its relation to cat, an image it has seen) from reading text.
It's not intelligent, not even close. But it's an awfully strange emergent phenomenon these concepts are demonstrating. Exciting stuff, I think.
It seems like the implicit target in the document is to achieve a critically damped system with no ringdown on learning. However, if they're trying to go for speed, then it seems like they should accept possible overshoot, and use non-linear control theory for their weights so that they're underdamped during the initial descent, and then transition into critically damped gradient descent as they move into the flat zo…
It seems like the implicit target in the document is to achieve a critically damped system with no ringdown on learning. However, if they're trying to go for speed, then it seems like they should accept possible overshoot, and use non-linear control theory for their weights so that they're underdamped during the initial descent, and then transition into critically damped gradient descent as they move into the flat zo…
That sort of sounds like Rprop which is supposedly the fastest learning algorithm. There are also other "adaptive learning rate" algorithms. https://en.wikipedia.org/wiki/Rprop