Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural networks in the 1990s

twitter.com

Post reply on HN
Print view View original

71–80 of 89 posts

Re: Neural networks in the 1990s

#71

Post by m-i-l »

bvan wrote: post #27

Yikes, I’m old. There was a lot of NN work and a lot of books available on NN’s back in the mid and late 90’s. ‘Soft computing’ was the all-encompassing term for NN, genetic algorithms, AI, expert systems, fuzzy logic, ALife and all sorts of nascent computational areas back then. I still have a bunch of issues to the monthly AI Expert magazine one could buy at a decent magazine stand. Small data-sets were definitely…

>"Small data-sets were definitely a limiting factor as well as limited computer power."

Not just small data-sets and limited computer power, but also very few libraries to help you out - although you could download something like xerion from ftp.cs.toronto.edu and join their email list, it was generally a case of retyping examples or implementing algorithms from printed textbooks. And it was all in C, presumably for performance reasons, while most of the symbolic AI folks came from Lisp or Prolog backgrounds.

Re: Neural networks in the 1990s

#72

Post by bravura »

sigmoid10 wrote: post #61
radq wrote: post #54

We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization. [1] Residual nets (2015): https://arxiv.org/abs/1512.03385 [2] Batch normalization (2015): https://arxiv.org/abs/1502.03167

Also quasi-linear activation functions (prevent vanishing gradients), tons of regularisation (e.g convolutions) and more adaptive gradient descent (faster convergence). I've still met people in the early 2010s who tried to make neural networks work using only a few dozen units. Academia is pretty slow. What people also forget is that libraries like pytorch or tensorflow simply didn't exist. I wrote my own neural netw…

LeCun et al (1989) had backprop working for digit recognition.

LeCun, Bottou, et al (2002) in "Efficient Backprop" described techniques for improving backprop algorithms.

Re: Neural networks in the 1990s

#73

Post by yobbo »

To experiment with SGD and back-propagation with 4096x4096 32-bit matrices, you would need a machine with hundreds of megabytes of ram in the 90s. In terms of software, you would need to be comfortable with C/C++ or maybe Fortran to be able to experiment quickly enough to land on effective hyper parameters.

Probably too many low-probability events chained together.

But I think they discovered most of the interesting things that small networks can do? For example, TD-Gammon from 1992: https://en.wikipedia.org/wiki/TD-Gammon .

Re: Neural networks in the 1990s

#76

Post by sigmoid10 »

bravura wrote: post #72
sigmoid10 wrote: post #61

Earlier quoted context omitted.

Also quasi-linear activation functions (prevent vanishing gradients), tons of regularisation (e.g convolutions) and more adaptive gradient descent (faster convergence). I've still met people in the early 2010s who tried to make neural networks work using only a few dozen units. Academia is pretty slow. What people also forget is that libraries like pytorch or tensorflow simply didn't exist. I wrote my own neural netw…

LeCun et al (1989) had backprop working for digit recognition. LeCun, Bottou, et al (2002) in "Efficient Backprop" described techniques for improving backprop algorithms.

Rosenblatt had a working perceptron for classifying images in the 1950s (!). And yet it took 60 years before the theory and compute power had developed enough for all of this to be interesting outside of small, purely academic experiments.

Re: Neural networks in the 1990s

#79

Post by ttul »

Ono-Sendai wrote: post #37
ttul wrote: post #30

In 1999, our “computer vision” guy - a masters student - struggled mightily to recognize very simple things in a video stream from a UAV. Today, we would take this for granted. But back then, the computation was for all intents and purposes entirely non-existent. At best he was hoping to apply an edge detection kernel maybe once every two seconds and see if he could identify some lines and arcs and then hand code som…

What? There were pentium 2 and 3 machines back then that could certainly do more than a edge detection kernel every 2 seconds. Or do you mean on an embedded CPU?

Indeed, this had to happen onboard a small airplane. Should have clarified that.

Re: Neural networks in the 1990s

#80

Post by amichal »

I followed a scientific American article in 1992 as a high schooler and got digit recognition and basic arithmetic working on a 386. What the popsci press said at the time was that we were limited by memory bandwidth (cache size), training data, and to some extend pointer-chasing (and other inefficencies) in graph algos
Post reply on HN