131–140 of 190 posts
Technically, he is right. But OpenAI was the first to find the winning formula and make it famous.
What do you mean by "winning formula" in this sentence?
For me personally: I have had success asking for code snippets and general brainstorming in areas which are not my forte, all the while using a very nice and clean UI.
Earlier quoted context omitted.
What would you change to the UX and design to make it even better?
Entering and editing text on a small touchscreen with it is a pretty bad experience. The keyboard covers the text, linebreaks can't be entered without pasting, viewing the output requires closing the keyboard. None of these are problems to the same degree in say Discord's mobile/tablet UI.
Earlier quoted context omitted.
> OpenAI's really interesting approach to GPT That's the issue though, Yann LeCun is specifically referring to ChatGPT as the standalone model, not the GPT family since a lot of models at Meta, Google, DeepMind are based on a similar approach. His point is that ChatGPT is a cosmetic additional training with prompt with a nice interface, but not a fundamentally different model than stuff we've have had for +2-3 years…
ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: https://arxiv.org/abs/2005.14165 It was a big deal in the NLP space. It wasn't a 'cosmetic' improvement over existing NLP approaches.
That's the point LeCunn is making. He's not out there negating that the paper you linked was ground-breaking, he's saying that converting that model into ChatGPT was not ground-breaking from an academic standpoint.
Earlier quoted context omitted.
Yes, but along that reasoning the criticism that "ChatGPT is not particularly innovative" is a bit disingenuous in the sense that the general public has never had easy access to these models until ChatGPT arrived. Besides, even though the newer GPT models presumably just extended earlier ones with similar techniques but better hardware and data, it wasn't a foregone conclusion how powerful (from the user perspective)…
I don't think the point here is to downplay the significance of ChatGPT as a product, but to point out that OpenAI is likely not uniquely resourced in this regard. If anything, it sounds like what they're trying to say is that ChatGPT is only the tip of the iceberg, and the reason why it came out of OpenAI and not Meta or Google is more a function of other factors like reputational damage, etc. The headline does hit…
Earlier quoted context omitted.
Please don't put in quotes something that is not a direct quote.
> > I would flippantly summarize them as: "..." > Please don't put in quotes something that is not a direct quote. What part of "I would flippantly summarize them as" was unclear? Or was that comment edited?
Earlier quoted context omitted.
> OpenAI's really interesting approach to GPT That's the issue though, Yann LeCun is specifically referring to ChatGPT as the standalone model, not the GPT family since a lot of models at Meta, Google, DeepMind are based on a similar approach. His point is that ChatGPT is a cosmetic additional training with prompt with a nice interface, but not a fundamentally different model than stuff we've have had for +2-3 years…
ChatGPT is build on GPT-3. GPT-3 was a big NLP development. The paper has 7000+ citations: https://arxiv.org/abs/2005.14165 It was a big deal in the NLP space. It wasn't a 'cosmetic' improvement over existing NLP approaches.
A tweet isn't worth a news article. Geeze.
That seems to be a lot of news articles these days. I'm not particularly excited by ChatGPT, but the mercenary in me is impressed at the investment from MS it generated. From a business perspective, it accomplished "and profit!". Whether it pays dividends or reaches a richer/more-sophisticated milestone is to be seen. Personally, I think it'll muddy and flood the "bullshit" & truthiness layer pervading news, social m…
He's not wrong. The key ingredient in ChatGPT is not brilliance but capital. It takes a lot of money not just to do the raw training but to get all the data together and process everything. There's no break through insights that make ChatGPT work, just a lot of consolidated wealth. The hacker part of me finds AI less and less interesting for this reason. We're seeing what the limits of pouring resources into the prob…
Normally it wouldn't, but we add an extra ingredient here. We get a validation signal. This is problem specific, but for code it would mean to integrate the LM with a compiler and runtime so it can iterate until it solves the task, step by step. For other tasks it could mean hooking the AI to simulators, games and robots to solve tasks. It is also possible to use LLMs as simulators of text.
Basically doing Reinforcement Learning with a Language Model and not just for human preferences, but for problem solving on a grand scale. Collect data from problem solving, train on it, and iterate. It costs just electricity, but LLMs can make their own data. Anthropic's Constitutional AI which is RLAIF - reinforcement learning from AI feedback is proof it can be done.