Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural networks in the 1990s

twitter.com

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Neural networks in the 1990s

#61

Post by sigmoid10 »

radq wrote: post #54

We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization. [1] Residual nets (2015): https://arxiv.org/abs/1512.03385 [2] Batch normalization (2015): https://arxiv.org/abs/1502.03167

Also quasi-linear activation functions (prevent vanishing gradients), tons of regularisation (e.g convolutions) and more adaptive gradient descent (faster convergence). I've still met people in the early 2010s who tried to make neural networks work using only a few dozen units. Academia is pretty slow. What people also forget is that libraries like pytorch or tensorflow simply didn't exist. I wrote my own neural network stacks complete with backpropagation from scratch in c++ back then.

Re: Neural networks in the 1990s

#62

Post by throwaway2037 »

CorrectHorseBat wrote: post #34
moomoo11 wrote: post #24

Earlier quoted context omitted.

encyclopedia Britannia existed. I came to USA in late 90s and my school had the CD set.

Wikipedia is ~100 bigger than the Encyclopædia Britannica https://en.m.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

Nice link! I never saw that page before. This quote surprised me:

    it should be noted that the amount of text added to Wikipedia articles every year has been constant since 2006, at roughly 1 gigabyte of (compressed) text added per year.

Re: Neural networks in the 1990s

#63

Post by nurbsnn »

Solvency wrote: post #8
brrrrrm wrote: post #3

I doubt it was obvious scaling up would magically work. I suspect the experiments were limited for analytic simplicity rather than computational.

Why couldn't mathematical proofs/models have predicted or revealed this to be the case back then?

On the contrary, there was a mathematical proof that one-hidden-layer neural network with nonlinearity is enough to represent any function. Using more than 1 hidden seemed a waste.

https://en.wikipedia.org/wiki/Universal_approximation_theore...

Re: Neural networks in the 1990s

#64

Post by mlajtos »

version_five wrote: post #6

I think it's more that modern automatic differentiation abstractions weren't well known to researchers. From what I remember, even in the early 2000s when I went to school, backpropagation was basically hand coded.

Yes, everything was hand coded (no autodiff) & Hinton loves Matlab.

Re: Neural networks in the 1990s

#65

Post by tyre »

throwaway2037 wrote: post #62
CorrectHorseBat wrote: post #34

Earlier quoted context omitted.

Wikipedia is ~100 bigger than the Encyclopædia Britannica https://en.m.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia

Nice link! I never saw that page before. This quote surprised me: it should be noted that the amount of text added to Wikipedia articles every year has been constant since 2006, at roughly 1 gigabyte of (compressed) text added per year.

Yes, Wikipedia is surprisingly small. You can fit the whole thing on an iPad and access all of it without internet. Plenty of rabbit holes to fill even the longest airplane flight.

Re: Neural networks in the 1990s

#66

Post by peterfirefly »

pavon wrote: post #17
brrrrrm wrote: post #3

I doubt it was obvious scaling up would magically work. I suspect the experiments were limited for analytic simplicity rather than computational.

The only ML that I ever did was a single undergrad NN class around ~2001. That was a long time ago, but I vaguely remember being taught at that time that adding more nodes rarely helped, that you were just going to overfit to your dataset and have worse results on items outside the dataset, or worse end up with a completely degenerate NN - eg that best practice was to use the minimum number of nodes that would do the…

"Optimal Brain Damage", 1989:

https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...

Re: Neural networks in the 1990s

#67

Post by peterfirefly »

FrustratedMonky wrote: post #28

Reading through the twitter thread, and these comments. It reminds me of all of the back and forth when HN discusses Psychology. One side, holding a pipe, 'well actually, back in 1954, I put together an analog variant of a neuron perceptron built out of old speaker cables and car parts, strung it across the living room and it could say 10 words and fetch my slippers'. 'Really', 'Yes, Indubitably'. The other side, It'…

https://en.wikipedia.org/wiki/Elmer_and_Elsie_(robots)

"Elmer and Elsie, or the "tortoises" as they were known, were constructed between 1948 and 1949 using war surplus materials and old alarm clocks."

"The robots were designed to show the interaction between both light-sensitive and touch-sensitive control mechanisms which were basically two nerve cells with visual and tactile inputs."

Re: Neural networks in the 1990s

#68

Post by arketyp »

radq wrote: post #54

We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization. [1] Residual nets (2015): https://arxiv.org/abs/1512.03385 [2] Batch normalization (2015): https://arxiv.org/abs/1502.03167

AlexNet predated that though.

Re: Neural networks in the 1990s

#69

Post by arketyp »

jjtheblunt wrote: post #48
Solvency wrote: post #7

Do you think Carmack, deep down, wonders why he let himself miss the boat on the LLM revolution? He spent golden years toiling away in Facebook, only to finally announce he was quitting to focus on AGI... only for the world to be taken by storm by transformers, GPT, Midjourney, etc. If anyone could have been at the forefront of this wave, it could've been him. And now the landscape has utterly changed and no one is e…

> If anyone could have been at the forefront of this wave, it could've been him. Why does DOOM and clever programming on a NeXT imply what you assert?

Indeed. Carmack wasn't simply betting on VR; more specifically, he was working on VR for mobile devices, where his expertise was a good fit.

Re: Neural networks in the 1990s

#70

Post by FrustratedMonky »

FrustratedMonky wrote: post #28

Reading through the twitter thread, and these comments. It reminds me of all of the back and forth when HN discusses Psychology. One side, holding a pipe, 'well actually, back in 1954, I put together an analog variant of a neuron perceptron built out of old speaker cables and car parts, strung it across the living room and it could say 10 words and fetch my slippers'. 'Really', 'Yes, Indubitably'. The other side, It'…

A very bad comment, that failed to make a point, and that wasn't very humorous.

I meant to make relationship between Psychology and Machine Learning.

Psychology, the study of the mind, with questionable scientific methods and a replication problem.

And

Machine Learning, (that is taking the mind as a model), with questionable scientific methods, and replication problem, and the addition of corporate hype machines.

Often in last few months we stand in awe of what AI achieves, but it produces questionable results, and has a lot of problems. Machine learning is worshiped.

And yet often in last few months, posts on Psychology is railed on and called a field full of con-men and BS-Artists.

Why the duality? Both are young fields and stretching. Rapidly making progress, hitting dead ends, and changing course. The scientific method isn't a strait path. But Psychology doesn't seem to be given much leeway to make errors and course correct.

I just find it hitting a peak right now, because the study of the Human Mind (wet net) and Machine Mind (electric net). Seem to be hitting a lot of the same issues. There are so many parallels in how they are spoken of, so many common problems and how they are framed within each field.

Wonder how long until we just openly talk about a field of Psychology of Machines, where we use the same tools to try and understand what the Neural Nets are thinking.

Post reply on HN