Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Basic Neural Network on Python

danielfrg.github.io

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Basic Neural Network on Python

#11

Post by stiff »

sine_dicendo wrote: post #10
benhamner wrote: post #3

Both datasets you used (iris and digits) are way too simple for neural networks to shine. Neural networks / deep neural networks work best in domains where the underlying data has a very rich, complex, and hierarchical structure (such as computer vision and speech recognition). Currently, training these models is both computationally expensive and fickle. Most state of the art research in this area is performed on GP…

Not for nothing but Ben did you read the article? He's not even discussing most of what you mention. He is simply taking his learning and applying it. You seem to be going off on a tangent about advanced applications where he is obviously just learning about how these things work and not trying to teach a method or suggesting that he has discovered anything significant.. To the author: I liked the article. A simple,…

He just shared some insights, he didn't critique anything and he's a Kaggler and published researcher so I don't get why he is getting downvoted.

Re: Basic Neural Network on Python

#12

Post by dbecker »

sine_dicendo wrote: post #10
benhamner wrote: post #3

Both datasets you used (iris and digits) are way too simple for neural networks to shine. Neural networks / deep neural networks work best in domains where the underlying data has a very rich, complex, and hierarchical structure (such as computer vision and speech recognition). Currently, training these models is both computationally expensive and fickle. Most state of the art research in this area is performed on GP…

Not for nothing but Ben did you read the article? He's not even discussing most of what you mention. He is simply taking his learning and applying it. You seem to be going off on a tangent about advanced applications where he is obviously just learning about how these things work and not trying to teach a method or suggesting that he has discovered anything significant.. To the author: I liked the article. A simple,…

In Ben's defense: The original article declares random forest a "winner" over neural networks. Ben's comment is a cautionary note that this result only applies to a specific class of problems.

This was a nice post, but it's reasonable to warn users not to overgeneralize the algorithm comparison.

Re: Basic Neural Network on Python

#13

Post by gamegoblin »

lightcatcher wrote: post #6
gamegoblin wrote: post #2

Very good write up. If you want to trade speed and memory for accuracy, you can make a large lookup table for your sigmoidal function which should just about double the speed of it. As an aside, and not to be too critical, because the post was great, but as (presumably) a non-native English speaker, you might do a spell-checker on your post. There are also some missing pronouns which make some sentences very Spanishy…

A relatively small lookup table for the sigmoid function can also work well. Here are the various sigmoid approximations that Theano (a library used for deep learning research among other things) offers: http://deeplearning.net/software/theano/library/tensor/nnet/...

I usually use an array with a few thousand entries. In C this gives me a 2.5x speedup over the exact function with no important decrease in accuracy.

Re: Basic Neural Network on Python

#14

Post by theschreon »

You could try the following improvements to speed up neural network training:

- Resilient Propagation (RPROP), it significantly speeds up training for full batch learning: http://davinci.fmph.uniba.sk/~uhliarik4/recognition/resource...

- RMSProp, introduced by Geoffrey Hinton, also speeds up training but can also be used for mini-batch learning: https://class.coursera.org/neuralnets-2012-001/lecture/67 (sign up to view the video)

Please consider more datasets when benchmarking methods:

- MNIST ( 70k 28x28 pixel images of handwritten digits ): http://yann.lecun.com/exdb/mnist/ . There are several wrappers for Python on github.

- UCI Machine Learning Repository: http://archive.ics.uci.edu/ml/datasets.html

Re: Basic Neural Network on Python

#15

Post by mbq »

You are just doing a simple validation on a test set rather than cross-validation; the point of CV is to make many iterations of validation on different train-test splits and average the results.

Re: Basic Neural Network on Python

#16

Post by dfrodriguez143 »

theschreon wrote: post #14

You could try the following improvements to speed up neural network training: - Resilient Propagation (RPROP), it significantly speeds up training for full batch learning: http://davinci.fmph.uniba.sk/~uhliarik4/recognition/resource... - RMSProp, introduced by Geoffrey Hinton, also speeds up training but can also be used for mini-batch learning: https://class.coursera.org/neuralnets-2012-001/lecture/67 (sign up to vi…

Definitely a lot to read and improvements to make. I will probably do a more complete benchmark with more datasets on a later post.

Thanks for the suggestions.

Re: Basic Neural Network on Python

#17

Post by dfrodriguez143 »

mbq wrote: post #15

You are just doing a simple validation on a test set rather than cross-validation; the point of CV is to make many iterations of validation on different train-test splits and average the results.

I agree completely a more complex benchmark should be done with a complete cross-validation.

Just for future reference I did ran the fitting a few times founding very(+-2%) similar results. Also Random Forests do an average so probably not much to improve on that particular algorithm.

Re: Basic Neural Network on Python

#18

Post by dfrodriguez143 »

gamegoblin wrote: post #13
lightcatcher wrote: post #6

Earlier quoted context omitted.

A relatively small lookup table for the sigmoid function can also work well. Here are the various sigmoid approximations that Theano (a library used for deep learning research among other things) offers: http://deeplearning.net/software/theano/library/tensor/nnet/...

I usually use an array with a few thousand entries. In C this gives me a 2.5x speedup over the exact function with no important decrease in accuracy.

Very interesting and simple improvement. Definitely will take a look at that.

Re: Basic Neural Network on Python

#19

Post by gallamine »

gamegoblin wrote: post #2

Very good write up. If you want to trade speed and memory for accuracy, you can make a large lookup table for your sigmoidal function which should just about double the speed of it. As an aside, and not to be too critical, because the post was great, but as (presumably) a non-native English speaker, you might do a spell-checker on your post. There are also some missing pronouns which make some sentences very Spanishy…

You could also try the Elliot Sigmoid activation function. I found it executed about 2x faster than the exponential sigmoid (in Matlab). Brief writeup: http://www.gallamine.com/blog/2013/01/21/a-sigmoid-function-...

Re: Basic Neural Network on Python

#20

Post by mistercow »

gamegoblin wrote: post #13
lightcatcher wrote: post #6

Earlier quoted context omitted.

A relatively small lookup table for the sigmoid function can also work well. Here are the various sigmoid approximations that Theano (a library used for deep learning research among other things) offers: http://deeplearning.net/software/theano/library/tensor/nnet/...

I usually use an array with a few thousand entries. In C this gives me a 2.5x speedup over the exact function with no important decrease in accuracy.

I wonder though if you might actually do better overall with a smaller lookup table and interpolation (or even just a polynomial approximation, which can be evaluated without branching), since large lookup tables can lead to bad cache behavior.
Post reply on HN