Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural networks in the 1990s

twitter.com

Post reply on HN
Print view View original

51–60 of 89 posts

Re: Neural networks in the 1990s

#51

Post by lyu07282 »

Solvency wrote: post #7

Do you think Carmack, deep down, wonders why he let himself miss the boat on the LLM revolution? He spent golden years toiling away in Facebook, only to finally announce he was quitting to focus on AGI... only for the world to be taken by storm by transformers, GPT, Midjourney, etc. If anyone could have been at the forefront of this wave, it could've been him. And now the landscape has utterly changed and no one is e…

I sometimes wonder what could've happened if he stuck to the 3d graphics space. He once was a great innovator, wolfenstein, doom then quake, he did some innovation in Rage / id Tech 5 with infinite texture streaming but it was full of technical issues. Ultimately around doom 3 / rage, it felt like id software wasn't anything special anymore, they were brought out and then he left Id.

Now the last major innovation in the space came from epic games / unreal engine.

Re: Neural networks in the 1990s

#53

Post by WithinReason »

lyu07282 wrote: post #51
Solvency wrote: post #7

Do you think Carmack, deep down, wonders why he let himself miss the boat on the LLM revolution? He spent golden years toiling away in Facebook, only to finally announce he was quitting to focus on AGI... only for the world to be taken by storm by transformers, GPT, Midjourney, etc. If anyone could have been at the forefront of this wave, it could've been him. And now the landscape has utterly changed and no one is e…

I sometimes wonder what could've happened if he stuck to the 3d graphics space. He once was a great innovator, wolfenstein, doom then quake, he did some innovation in Rage / id Tech 5 with infinite texture streaming but it was full of technical issues. Ultimately around doom 3 / rage, it felt like id software wasn't anything special anymore, they were brought out and then he left Id. Now the last major innovation in…

He did his best work when he wrote the entire engine alone. That's no longer possible. You can however plausibly invent AGI alone. He said that an AGI implementation is likely simple (meaning not complex), and I agree. The difficulty is in the method not lines of code, so it's work that fits him.

Re: Neural networks in the 1990s

#54

Post by radq »

We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization.

[1] Residual nets (2015): https://arxiv.org/abs/1512.03385

[2] Batch normalization (2015): https://arxiv.org/abs/1502.03167

Re: Neural networks in the 1990s

#55

Post by rwmj »

I knew someone in the early 90s who was making a neural network on a chip for his PhD. The chip fitted 1 neuron. Yes he might have used float16 to cram more in but those techniques were not known at the time.

There really wasn't the compute power around at the time, and as others have pointed out there wasn't the training data, or the cameras.

Re: Neural networks in the 1990s

#56

Post by dunefox »

robg wrote: post #4

Highly recommend the exercises in Rumelhart and McClelland - Parallel Distributed Processing: Explorations in the Microstructure of Cognition from 1986-1987 (two volumes) https://direct.mit.edu/books/book/4424/Parallel-Distributed-...

Does it hold up for today?

Re: Neural networks in the 1990s

#58

Post by l33tman »

shagie wrote: post #44
Palomides wrote: post #23

Earlier quoted context omitted.

is that really true? a modern high end GPU has more computing power than the top 20 supercomputers of the year 2000 added together

My favorite comparison for the accessibility of power is looking at a weird computer in the top 500 from a while back. System X, in 2004 was the 7th most powerful computer in the world. It was 1100 PowerPC 970 Macs with 2200 cores and claimed an Rmax of 12k GFlops. https://www.top500.org/system/173736/ A M1 MacBook Air hits 900 Gflops ( https://news.ycombinator.com/item?id=26333369 ). A dozen MacBook Airs - about wha…

The RTX 4090 GPU (a single PCI card) hits 82k GFlops in FP32

Re: Neural networks in the 1990s

#59

Post by hzay »

radq wrote: post #54

We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization. [1] Residual nets (2015): https://arxiv.org/abs/1512.03385 [2] Batch normalization (2015): https://arxiv.org/abs/1502.03167

Yes, but the tweet is talking about single layer networks!

Re: Neural networks in the 1990s

#60

Post by 2sk21 »

I have one of the early PhDs in neural networks (graduated in 1992). However my work was analytical - I was able to prove a couple of theorems about the backpropagation. I just needed a simple implementation to prove that my ideas worked so I wrote my code from scratch in C.
Post reply on HN