Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Basic Neural Network on Python

danielfrg.github.io

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Basic Neural Network on Python

#21

Post by jph00 »

benhamner wrote: post #3

Both datasets you used (iris and digits) are way too simple for neural networks to shine. Neural networks / deep neural networks work best in domains where the underlying data has a very rich, complex, and hierarchical structure (such as computer vision and speech recognition). Currently, training these models is both computationally expensive and fickle. Most state of the art research in this area is performed on GP…

Handwritten digits actually is a pretty good domain for deep nets, and the poor performance achieved in the article's case is due to the implementation (it needs deeper net, convolutional layer, etc). In that case much better (99%+) results have been achieved by deep nets for digit recognition. In fact, Hinton (in his Coursera course) recommends this domain for studying deep nets, since it is so well understood.

(Ben I know you're aware of all this already, but I just wanted to clarify for those who aren't as on top of the research as you)

Re: Basic Neural Network on Python

#22

Post by mbq »

dfrodriguez143 wrote: post #17
mbq wrote: post #15

You are just doing a simple validation on a test set rather than cross-validation; the point of CV is to make many iterations of validation on different train-test splits and average the results.

I agree completely a more complex benchmark should be done with a complete cross-validation. Just for future reference I did ran the fitting a few times founding very(+-2%) similar results. Also Random Forests do an average so probably not much to improve on that particular algorithm.

To be honest I don't expect the results to change; but this is an only way to attach significance to the observed differences and to ensure this wasn't a lucky shot.

Re: Basic Neural Network on Python

#25

Post by benhamner »

dfrodriguez143 wrote: post #16
theschreon wrote: post #14

You could try the following improvements to speed up neural network training: - Resilient Propagation (RPROP), it significantly speeds up training for full batch learning: http://davinci.fmph.uniba.sk/~uhliarik4/recognition/resource... - RMSProp, introduced by Geoffrey Hinton, also speeds up training but can also be used for mini-batch learning: https://class.coursera.org/neuralnets-2012-001/lecture/67 (sign up to vi…

Definitely a lot to read and improvements to make. I will probably do a more complete benchmark with more datasets on a later post. Thanks for the suggestions.

You may be interested in this ICML 2006 paper, which empirically compared many standard algorithms across a combination of metrics and UCI datasets - http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icm...

Re: Basic Neural Network on Python

#26

Post by lelandbatey »

dfrodriguez143 wrote: post #24
lelandbatey wrote: post #23

Hmmmm... The layout of the page seems very messed up. Is anyone else having it show up like this?: http://puu.sh/3vTL8.png

Should work with most newer versions of any browser. Which browser are you using?

Firefox 22 on Windows 8

Re: Basic Neural Network on Python

#28

Post by wfn »

scotty79 wrote: post #27

What learning scientists think brain actually uses? Back-propagation and such seem like a method god would use to architect static brain for given task.

For starters - see Hebbian theory. [1]

Backprop falls within the class of 'supervised learning' which can indeed be said not to be very biologically realistic. However, reinforcement learning is observed, so the overall picture is probably much more complex: e.g. associative/recurrent/etc networks with Hebb-like unsupervised learning developing neuronal group testing and selection systems that involve reinforcement learning. (see first lecture/talk in [3].)

Perhaps worth a watch is a very nice talk by Geoffrey Hinton [2], which is oft referred to on HN. (Hinton does refer to the notion of biological plausibility etc. in this talk as far as I recall, but the focus is elsewhere (developing next generation state-of-the-art (mostly unsupervised) machine learning techniques/systems.))

[1]: https://en.wikipedia.org/wiki/Hebbian_theory

[2]: https://www.youtube.com/watch?v=AyzOUbkUf3M

[3]: http://kostas.mkj.lt/almaden2006/agenda.shtml (The original summary HTML file is gone from the original source, so this is a mirror; the links to videos and slides do work, though.) The first and the second talks are somewhat relevant (particularly the first one, re: bio plausibility etc ("Nobelist Gerald Edelman, The Neurosciences Institute: From Brain Dynamics to Consciousness: A Prelude to the Future of Brain-Based Devices")), but all are great. Rather heavy, though. (Also, skip the intros.)

edit that first talk/lecture from Almaden (Edelman's) is actually a very nice exposure of the whole paradigm in which {cognitive,computational,etc} neuroscience rests; it does get hairy later on; overall, it's a great talk for the truly curious.

Re: Basic Neural Network on Python

#29

Post by rullgrus »

lelandbatey wrote: post #23

Hmmmm... The layout of the page seems very messed up. Is anyone else having it show up like this?: http://puu.sh/3vTL8.png

Same here with Firefox 22.0. With 2560x1440 resolution you get three columns for the code blocks. It looks fine in IE 10. IE renders the page with only one column independent of window width.
Post reply on HN