51–60 of 97 posts
It’s interesting in that time we almost completely lost interest in neural networks and then came back around to them.
Earlier quoted context omitted.
30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then
My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be…
8% less growth is not the point. The "law" has stood the test of time which says something about the guy and his vision
Fine-tuning would then be used for specific environments (human hair, an apartment) and robots (robotic barber, cleaning robot).
Possible future development I'm excited about is something like GPT but for real world interaction - i.e. robots that take input from sensors and are able to physically navigate and manipulate the world. Fine-tuning would then be used for specific environments (human hair, an apartment) and robots (robotic barber, cleaning robot).
Earlier quoted context omitted.
This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.
It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)
The greatest form of general intelligence at 20W.
A MacBook Air is ~30W.
https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...
Maybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities. LeCun is not even really interested in supervised learning anymore, for example. https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY Note that the talk is not really primarily about ChatGPT even though that's in the title. The new…
I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and us…
Evolution has optimized animal brain and bodies to survive and take care of the next generation. They have a good grasp of environment, where they are, where food is, where predators are, basic communication if they live as a group. Babies grow up and start learning.
Our current AI is extremely power hungry compared to a brain. Cruise & Waymo put large power hungry supercomputers in cars. The computing system costs 100k+. They still make silly mistakes like crashing into fire trucks, blocking roads, driving into wet cement etc.
ChatGPT and friends make silly mistakes for trivial math problems that require a few hierarchical planning steps.
All in all, brains have some form of symbolic computation and reasoning that we haven’t been able to replicate with current AI algorithms.
I’m not saying we’ll never be able to but current AI is really hyped. Kinda like crypto boom of 2019.
There are some really hard algorithmic problems to be solved.
Google, Microsoft, Meta could have 1000x more computing power and data, however in the grand space or all algorithms there exists a learning algorithm that is probably >10000X more efficient at generalized modeling and reasoning than what we have.
The proof that we (20W biological generally intelligent computers) exist validates the hypothesis that there is a lot of advancement we can still do at the algorithm part.
Remember, 33 years ago "connectionist AI" wasn't the dominant AI paradigm, and "symbolic AI" wasn't the only other approach either - there were others, like "robotic functionalism" (the idea that you couldn't have true intelligence with interacting with the physical world). Maybe in 33 years some of these other approaches will have a resurgence, perhaps in combination with connectionist approaches. Or maybe they'll even be some entirely new approach.
Maybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities. LeCun is not even really interested in supervised learning anymore, for example. https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY Note that the talk is not really primarily about ChatGPT even though that's in the title. The new…
This is exactly why I didn't start experimenting with this stuff back then. I read some articles and had the interest, but having no access to existing training data or "fast" computers was really a show stopper. This article really convinced me that the amazing results today are mostly due to hardware advances.
I will add my own view that 1) hardware will not be advancing anywhere near so much in the future. And 2) training and inference have to be done together like real brains do. Then the AI will learn from experience while deployed and you can clone the best ones later.