Sounds like bitter lemons to me from a team that's getting grief from Zuck for being shit and not getting there first or having the sense to.
"First they ignore you, then they laugh at you, then they fight you, then you win"
Yann LeCun: ChatGPT is 'not particularly innovative'
121–130 of 190 posts
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#122Earlier quoted context omitted.
> thought and heuristic patterns I’m not sure these are well-defined enough terms to make this claim.
Heuristics is the ability to recognize multiple options and reason which one is correct. ChatGPT (and all GPT models for that matter) have a single context that doesn't get compared to other possible generations. You can back-propegate and see which tokens influenced the output most, but there is no evidence that AI can conceive multiple hypothetical responses and select the most-preferred one. At least not ChatGPT.
Have you heard of "beam search"? The model keeps a number of optional lines of text without being greedy and picking only the most probable token every time. You can also sample with T>0 multiple times to get an ensemble of answers that can be used as input for another AI operation. You just need to do multiple interactions with the model.
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#123Yann's tweets on this are pretty funny. I would flippantly summarize them as: "Don't get me wrong, the openai people are great. All I'm doing is educating the media, who have the mistaken belief that chatGPT is good" https://twitter.com/ylecun/status/1617921903934726144?t=2wZZ...
Please don't put in quotes something that is not a direct quote.
> > I would flippantly summarize them as: "..."
> Please don't put in quotes something that is not a direct quote.
What part of "I would flippantly summarize them as" was unclear? Or was that comment edited?
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#124Earlier quoted context omitted.
Clarke's third law: "Any sufficiently advanced technology is indistinguishable from magic." Most people don't understand that ChatGPT has no idea what they're talking about. It lacks it's own thought and heuristic patterns, and only gives you the most-likely response to your prompt. People don't know that though, so they think the Mechanical Turk is actually playing chess. I mostly agree with the headline here. ChatG…
"hardly any more innovative than a Markov chain." Innovation is not invention. Innovation includes combining pedestrian concepts in novel ways. ChatGPT has the best ux and richest corpus of any markov chain I've ever used. It fits my bill of innovation combining sevaral things into a dang appealing product.
Whatever they want to call it, it is here and it works.
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#125It's amazing how many researchers underestimate the importance of UX and design. Personally, seeing non-technical people using prompts for the first time -- and getting results that make sense -- is so incredible. Their eyes light up, they are surprised, and they want to keep playing with it. Amazing! A sizeable part of the population just got access to an incredibly complex AI model and can play with it. What a coll…
I don't think he's underestimating the UX/design piece, I think it's just outside of the scope of what he's talking about. It seems pretty clear to me that his statement is talking about the underlying AI technology (which makes sense given his background). And in any case, I wouldn't call anything about the UX or design here innovative - it's obviously a huge improvement in terms of usability, but a chat UI is a pre…
Nobody talks about the dataset. Yes, the model was not innovative. But why hasn't anyone equaled OpenAI yet? Maybe they innovated on data engineering.
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#126Earlier quoted context omitted.
Researchers point of view is based on their area of research and that's fair and expected. Yann LeCun compares ChatGPT in the context of the related research. Imagine a ChatGPT equivalent that memorizes many questions and does a brute force strategy for an answer. It may "look" magic, but there's nothing magic about it. We all accepted that this is the case with Blue Gene - https://en.wikipedia.org/wiki/Deep_Blue_(ch…
But ChatGPT does not use brute force search to look for an answer. It interpolates among the answers in its training set. I.e. in Yann LeCun's analogy of a cake, interpolation or unsupervised learning is the cake; direct feedback on each single data point or supervised learning is the icing on the cake; and general feedback of the "how I am doing" sort or reinforcement learning is the cherry on top. Now LeCun is just…
He is saying "chatGPT is mostly cake and cake wasn't invented here".
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#127Earlier quoted context omitted.
It's because ChatGPT's knowledge doesn't come through interaction with the world. Words mean things to us because they point to world interactions. I saw a demo where ChatGPT could create a VM. It could be trained to interact directly with a VM, send commands to an interpreter. In this case, it would understand the response of the VM, although it wouldn't understand the design behind the VM, because humans did that b…
It did, actually. The model was trained with multiple rounds of reinforcement learning where human judges provided the feedback: first with full answers, and then with ranking of answers as most relevant. So the model in production is probably frozen, but before that it went through multiple rounds of interaction with the world.
The reinforcement learning was on giving the right answer, not on interacting with the world. But there is movement in the right direction with https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-...
and other RL stuff. (RT-1 isn't RL but there is other related stuff that is)
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#128Earlier quoted context omitted.
OK, I'll defend the research, too. OpenAI's really interesting approach to GPT was to scale the size of the underlying neural network. They noticed that the performance of an LLM kept improving as the size of the network grew so they said, "Screw it, how about if we make it have 100+ billion parameters?" Turns out they were right. From a research perspective, I'd say this was a big risk and it turned out they were ri…
> OpenAI's really interesting approach to GPT That's the issue though, Yann LeCun is specifically referring to ChatGPT as the standalone model, not the GPT family since a lot of models at Meta, Google, DeepMind are based on a similar approach. His point is that ChatGPT is a cosmetic additional training with prompt with a nice interface, but not a fundamentally different model than stuff we've have had for +2-3 years…
ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: https://arxiv.org/abs/2005.14165 It was a big deal in the NLP space.
It wasn't a 'cosmetic' improvement over existing NLP approaches.
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#129It's amazing how many researchers underestimate the importance of UX and design. Personally, seeing non-technical people using prompts for the first time -- and getting results that make sense -- is so incredible. Their eyes light up, they are surprised, and they want to keep playing with it. Amazing! A sizeable part of the population just got access to an incredibly complex AI model and can play with it. What a coll…
>> using prompts for the first time -- and getting results that make sense -- is so incredible. Years ago it was possible to insert prompts into Google and get results that made sense, results that were meaningfully tied to the information you requested. The young people today who missed that time think it magic.
That wonder might not be lost forever. Can't we have access to an old index and cached web pages?
Re: Yann LeCun: ChatGPT is 'not particularly innovative'
#130Earlier quoted context omitted.
It's because ChatGPT's knowledge doesn't come through interaction with the world. Words mean things to us because they point to world interactions. I saw a demo where ChatGPT could create a VM. It could be trained to interact directly with a VM, send commands to an interpreter. In this case, it would understand the response of the VM, although it wouldn't understand the design behind the VM, because humans did that b…
Based on this alleged limitation, can you list tasks that you think AI's won't ever be able to succeed at?
I think we will see AGI. But for the AI to be robust, it has to interact with the world, even if it is a simulated one. We need to build an AI that knows what a toddler knows before we can build one that understands wikipedia.