Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural networks in the 1990s

twitter.com

Post reply on HN
Print view View original

31–40 of 89 posts

Re: Neural networks in the 1990s

#31

Post by WiSaGaN »

Palomides wrote: post #23
WiSaGaN wrote: post #11

I believe the issue was not a lack of computational power, but rather that people at the time didn't think large models with many parameters would effect meaningful change. This was even true three years ago, albeit on a different scale. As Ilya Sutskever expressed, people were not convinced there was still room to increase the scale. For the status quo to shift, two things could happen: a substantial reduction in co…

is that really true? a modern high end GPU has more computing power than the top 20 supercomputers of the year 2000 added together

Computers are undoubtedly more powerful now than they were in the 90s. Although computing capabilities of the 90s seem weak compared to today's standards, they were not so inadequate that we couldn't train and run a network comprising thousands of parameters. I vividly recall the early 2000s when I was in college. Neural networks were seen as a sort of "fringe" technology in a series of statistics courses. We were mostly shown examples with 6 or 12 neurons, and nobody mentioned the possibility of scaling up to hundreds of neurons. Around that time, we already had sophisticated games like The Elder Scrolls III. We could have easily scaled up the network size by at least an order of magnitude at home, not to mention the capabilities that big companies possessed at that time.

Re: Neural networks in the 1990s

#32

Post by rm999 »

While my experience is not from the 90s, I think I can speak to some of why this is. For some context, I first got into neural networks in the early 2000s during my undergrad research, and my first job (mid 2000s) was at an early pioneer that developed their V1 neural network models in the 90s (there is a good chance models I evolved from those V1 models influenced decisions that impacted you, however small).

* First off, there was no major issue with computation. Adding more units or more layers isn't that much more expensive. Vanishing gradients and poor regulation were a challenge and meant that increasing network size rarely improved performance empirically. This was a well known challenge up until the mid/later 2000s.

* There was a major 'AI winter' going on in the 90s after neural networks failed to live up to their hype in the 80s. Computer vision and NLP researchers - fields that have most famously recently been benefiting from huge neural networks - largely abandoned neural networks in the 90s. My undergrad PI at a computer vision lab told me in no uncertain terms he had no interest in neural networks, but was happy to support my interest in them. My grad school advisors had similar takes.

* A lot of the problems that did benefit from neural networks in the 90s/early 2000s just needed a non-linear model, but did not need huge neural networks to do well. You can very roughly consider the first layer of a 2-layer neural network to be a series of classifiers, each tackling a different aspect of the problem (e.g. the first neuron of a spam model may activate if you have never received an email from the sender, the second if the sender is tagged as spam a lot, etc). These kinds of problems didn't need deep, large networks, and 10-50 neuron 2-layer networks were often more than enough to fully capture the complexity of the problem. Nowadays many practitioners would throw a GBM at problems like that and can get away with O(100) shallow trees, which isn't very different from what the small neural networks were doing back then.

Combined, what this means from a rough perspective, is that the researchers who really could have used larger neural networks abandoned them, and almost everyone else was fine with the small networks that were readily available. The recent surge in AI is being fueled by smarter approaches and more computation, but arguably much more importantly from a ton more data that the internet made available. That last point is the real story IMO.

Re: Neural networks in the 1990s

#33

Post by [deleted] »

WiSaGaN wrote: post #31
Palomides wrote: post #23

Earlier quoted context omitted.

is that really true? a modern high end GPU has more computing power than the top 20 supercomputers of the year 2000 added together

Computers are undoubtedly more powerful now than they were in the 90s. Although computing capabilities of the 90s seem weak compared to today's standards, they were not so inadequate that we couldn't train and run a network comprising thousands of parameters. I vividly recall the early 2000s when I was in college. Neural networks were seen as a sort of "fringe" technology in a series of statistics courses. We were mo…

[deleted]

Re: Neural networks in the 1990s

#34

Post by CorrectHorseBat »

moomoo11 wrote: post #24
MilStdJunkie wrote: post #14

Data, data, data, data. 1990s don't have wikipedia, Youtube, megapixel cameras every which where, every single adult human hooked up to a sensor package 24 hours a day, and who knows what else. I know as a 1990s guy I would never have imagined the amount of data we would eventually all throw up into the ether even ten years later, to say nothing of today. Without that corpus . .

encyclopedia Britannia existed. I came to USA in late 90s and my school had the CD set.

Wikipedia is ~100 bigger than the Encyclopædia Britannica

https://en.m.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

Re: Neural networks in the 1990s

#35

Post by p_l »

bvan wrote: post #27

Yikes, I’m old. There was a lot of NN work and a lot of books available on NN’s back in the mid and late 90’s. ‘Soft computing’ was the all-encompassing term for NN, genetic algorithms, AI, expert systems, fuzzy logic, ALife and all sorts of nascent computational areas back then. I still have a bunch of issues to the monthly AI Expert magazine one could buy at a decent magazine stand. Small data-sets were definitely…

Let's not forget that especially early 1990s are still in shock from AI Winter and there's essentially no funding.

Re: Neural networks in the 1990s

#36

Post by Ono-Sendai »

I personally made a quake 2 bot using neural networks in 1999, I think it had several hundred neurons and several thousand 'synapses' (parameters). At the time that felt like a lot of parameters. Computation wasn't much of a limit though, I could run several NNs faster than realtime.

Re: Neural networks in the 1990s

#37

Post by Ono-Sendai »

ttul wrote: post #30

In 1999, our “computer vision” guy - a masters student - struggled mightily to recognize very simple things in a video stream from a UAV. Today, we would take this for granted. But back then, the computation was for all intents and purposes entirely non-existent. At best he was hoping to apply an edge detection kernel maybe once every two seconds and see if he could identify some lines and arcs and then hand code som…

What? There were pentium 2 and 3 machines back then that could certainly do more than a edge detection kernel every 2 seconds. Or do you mean on an embedded CPU?

Re: Neural networks in the 1990s

#38

Post by adrianN »

WiSaGaN wrote: post #31
Palomides wrote: post #23

Earlier quoted context omitted.

is that really true? a modern high end GPU has more computing power than the top 20 supercomputers of the year 2000 added together

Computers are undoubtedly more powerful now than they were in the 90s. Although computing capabilities of the 90s seem weak compared to today's standards, they were not so inadequate that we couldn't train and run a network comprising thousands of parameters. I vividly recall the early 2000s when I was in college. Neural networks were seen as a sort of "fringe" technology in a series of statistics courses. We were mo…

Around fifteen years ago my machine learning professor dismissed neural networks because training them is not a nice convex optimization problem.

Re: Neural networks in the 1990s

#39

Post by bandrami »

Ono-Sendai wrote: post #37
ttul wrote: post #30

In 1999, our “computer vision” guy - a masters student - struggled mightily to recognize very simple things in a video stream from a UAV. Today, we would take this for granted. But back then, the computation was for all intents and purposes entirely non-existent. At best he was hoping to apply an edge detection kernel maybe once every two seconds and see if he could identify some lines and arcs and then hand code som…

What? There were pentium 2 and 3 machines back then that could certainly do more than a edge detection kernel every 2 seconds. Or do you mean on an embedded CPU?

Software is iterative. At least when I was studying in the mid 90s people had really only just gotten the idea to do a Fourier transform of an image and look for high frequencies to indicate borders. Add ~3 decades of each generation of grad student doing slightly better than the last one.

Re: Neural networks in the 1990s

#40

Post by watersb »

robg wrote: post #4

Highly recommend the exercises in Rumelhart and McClelland - Parallel Distributed Processing: Explorations in the Microstructure of Cognition from 1986-1987 (two volumes) https://direct.mit.edu/books/book/4424/Parallel-Distributed-...

I was studying computer science and AI in 1987-1990; I didn't know it was the deepest, darkest pit of AI research despair.

I found the two Rumelhart & McClelland books, just a single copy on the shelf at Cody's Books, soon after publication. I worked through the examples, and was immediately convinced that this low-level approach was a way forward.

For some reason, none of the stressed out Comp Sci professors wanted to listen to a weirdo undergraduate, a lousy student.

I'm glad I was there at a reboot of AI, but my timing was lousy.

Post reply on HN