Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural networks in the 1990s

twitter.com

Post reply on HN
Print view View original

81–89 of 89 posts

Re: Neural networks in the 1990s

#81

Post by shagie »

l33tman wrote: post #58
shagie wrote: post #44

Earlier quoted context omitted.

My favorite comparison for the accessibility of power is looking at a weird computer in the top 500 from a while back. System X, in 2004 was the 7th most powerful computer in the world. It was 1100 PowerPC 970 Macs with 2200 cores and claimed an Rmax of 12k GFlops. https://www.top500.org/system/173736/ A M1 MacBook Air hits 900 Gflops ( https://news.ycombinator.com/item?id=26333369 ). A dozen MacBook Airs - about wha…

The RTX 4090 GPU (a single PCI card) hits 82k GFlops in FP32

The reason I like the comparison (and the "here's this giant computer and now it fits on a card that you can get at Micro Center" is another reasonable comparison) is that it deals with likeish to likeish.

It was a Mac back then - 1100 of them, but it was a Mac. You could walk into a store and buy one... or two. They might have some issue with buying a thousand of them, but they were consumer commodity equipment - it was the rack mounted version of the PowerMac G5 if I read things correctly. You might have one of them in the media lab for a high school.

And now, it's a dozen M1 MacBook Airs (or Mac minis). Still a Mac. Still something you could walk into the store and buy. But now instead of "maybe there are 1000 of them in all the grade and high schools in the state" (though that would be stretching it), its "now this is an acceptably outfitted grade school computer lab."

No regular person was ever going to get proper fraction of the nodes of BlueGene from DOE (though it was running a PowerPC 440 2C instead... but 32,768 of them) or do anything with it if they were. https://en.wikipedia.org/wiki/IBM_Blue_Gene

Comparing "that massive thing" to "this card" is impressive - but the "that massive thing" is inconceivable to the average person.

Thus the "you could have gotten a fraction of System X at a store and used it" comparison.

Re: Neural networks in the 1990s

#83

Post by low_tech_love »

rm999 wrote: post #32

While my experience is not from the 90s, I think I can speak to some of why this is. For some context, I first got into neural networks in the early 2000s during my undergrad research, and my first job (mid 2000s) was at an early pioneer that developed their V1 neural network models in the 90s (there is a good chance models I evolved from those V1 models influenced decisions that impacted you, however small). * First…

The funny thing is that the authors of the paper he linked actually answer his question in the first paragraph, when they say that the input dataset needs to be significantly larger than the number of weights to achieve good generalisation, but there is usually not enough data available.

Re: Neural networks in the 1990s

#84

Post by rvba »

ldjkfkdsjnv wrote: post #29
reverius42 wrote: post #21

Earlier quoted context omitted.

And none of those examples except Wikipedia were used to train the various LLMs. I wonder how much better multi-modal models are going to get if they start incorporating the 24/7 sensor data from billions of people.

I cant wait for the time when someone trains a multimodal LLM on all of youtube

"Don't forget to like and subscribe"

On a side note, long time ago I saw someone who make a bot trained on a selected sample of chats between people on the internet - and the tool swore a lot.

Re: Neural networks in the 1990s

#85

Post by kristopolous »

WiSaGaN wrote: post #18
kristopolous wrote: post #15

Earlier quoted context omitted.

Did you post something nearly identical to this before? I feel like I read it before.

Are you referring to other threads? No. However, I wouldn't be surprised if other people developed similar beliefs following recent advances in large language models (LLMs). Of course, we wouldn't achieve GPT-4 level results using only technology available before 2020, but with sufficient data and computational power, we could have accomplished much more than what was generally believed to be possible in the machine…

I thought I had read almost those exact words before. I've been known to repeat myself on here before.

In fact I've been so nuanced that I've had people use something I've said to disagree with me and then I've had to point out that the original thing is also by me.

I've been unable to determine whether I'm actually influential or am just unknowingly expressing part of a generalized changing sentiment. Confidence is the first trapping of fools.

Re: Neural networks in the 1990s

#86

Post by r13a »

mjan22640 wrote: post #50

In 2012 were published results of a vision processing in the brain research, that (among other things, like the retina compressing the input) figured out that visual cortex uses convolution. That got mimicked and was a breakthrough in image recognition NN, which sparked life into the whole field.

Would you mind giving a reference to the paper? A quick googling didn't brought anything.

Re: Neural networks in the 1990s

#87

Post by l33tman »

shagie wrote: post #81
l33tman wrote: post #58

Earlier quoted context omitted.

The RTX 4090 GPU (a single PCI card) hits 82k GFlops in FP32

The reason I like the comparison (and the "here's this giant computer and now it fits on a card that you can get at Micro Center" is another reasonable comparison) is that it deals with likeish to likeish. It was a Mac back then - 1100 of them, but it was a Mac. You could walk into a store and buy one... or two. They might have some issue with buying a thousand of them, but they were consumer commodity equipment - it…

I see your logic, but the apple hardware is super-expensive, a single Macbook is the same cost as a single RTX 4090 (not the MBA maybe, but definitely the MBP). So it's not that wide a stretch to say that the 4090 in a normal PC is also a fair comparison as a "widely available" computer.

Re: Neural networks in the 1990s

#88

Post by bravura »

sigmoid10 wrote: post #76
bravura wrote: post #72

Earlier quoted context omitted.

LeCun et al (1989) had backprop working for digit recognition. LeCun, Bottou, et al (2002) in "Efficient Backprop" described techniques for improving backprop algorithms.

Rosenblatt had a working perceptron for classifying images in the 1950s (!). And yet it took 60 years before the theory and compute power had developed enough for all of this to be interesting outside of small, purely academic experiments.

Handwriting recognition on checks (LeCun et al 1989) wasn't really a small, purely academic experiment

Re: Neural networks in the 1990s

#89

Post by sigmoid10 »

bravura wrote: post #88
sigmoid10 wrote: post #76

Earlier quoted context omitted.

Rosenblatt had a working perceptron for classifying images in the 1950s (!). And yet it took 60 years before the theory and compute power had developed enough for all of this to be interesting outside of small, purely academic experiments.

Handwriting recognition on checks (LeCun et al 1989) wasn't really a small, purely academic experiment

And yet classical OCR techniques continued to dominate. Nothing happened in the industry on that front for over 20 years. That's as academic as it gets.
Post reply on HN