dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.
OpenAI’s CEO says the age of giant AI models is already over
191–200 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#192Earlier quoted context omitted.
Video. > YouTubers upload about 720,000 hours of fresh video content per day. Over 500 hours of video were uploaded to YouTube per minute in 2020, which equals 30,000 new video uploads per hour. Between 2014 and 2020, the number of video hours uploaded grew by about 40%.
But what are you mostly "teaching" the LLM then? Mundane everyday stuff? I guess that would make them better at "being average human" but is that what we want? It already seems that prompting the LLM to be above-average ("pretend to be an expert") improves performance.
This whole conversation about training set size is bizarre. No one ever asks what’s in the training set. Why would a trillion tokens of mundane gossip improve a LLMs ability to do anything valuable at all?
If a scrape of the general internet, scientific papers and books isn’t enough, a trillion trillion trillion text messages to mom aren’t going to change matters.
Re: OpenAI’s CEO says the age of giant AI models is already over
#193The most comforting AI news I have read this year.
If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild
I have repeatedly argued against this notion of „just predicting the next word“. No. It‘s completing a conversation. It‘s true that it is doing this word by word, but it‘s kind of like saying a CNN is just predicting a label. Sure, but how? It‘s not doing it directly. It‘s doing it by recovering a lot of structure and in the end boiling that down to a label. Likewise a network trained to predict the next word may very well have worked out the whole sentence (implicitly, not as a text) in order to generate the next word.
Re: OpenAI’s CEO says the age of giant AI models is already over
#194Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…
> model complexity necessary for spam filtering went down as the size of the data set went up
My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data.
Is there any merit to this belief?
Re: OpenAI’s CEO says the age of giant AI models is already over
#195Related reading: https://dynomight.net/scaling/ In short it seems like virtually all of the improvement in future AI models will come from better algorithms, with bigger and better data a distant second, and more parameters a distant third. Of course, this claim is itself internally inconsistent in that it assumes that new algorithms won't alter the returns to scale from more data or parameters. Maybe a more precise…
Good thing he got a bunch of companies to pony up the dough for LLM before he announced they where already over.
I don't think LLMs are over [0]. I think we're relatively close to a local optimum in terms of what can be achieved with current algorithms. But I think OpenAI is at least as likely as any other player to create the next paradigm, and that it's at least as likely as likely as any other player to develop the leading models within the next paradigm regardless of who actually publishes the research.
Separately, I think OpenAI's current investors have a >10% chance to hit the 100x cap on their returns. Their current models are already good enough to address lots of real-world problems that people will pay money to solve. So far they've been much more model-focused than product-focused, and by turning that dial toward the product side (as they did with ChatGPT) I think they could generate a lot of revenue relatively quickly.
[0] Except maybe in the sense that future models will be predominantly multimodal and therefore not strictly LLMs. I don't think that's what you're suggesting though.
Re: OpenAI’s CEO says the age of giant AI models is already over
#196Earlier quoted context omitted.
or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…
> model complexity necessary for spam filtering went down as the size of the data set went up My naive conclusion in that this means there are still massive gains to be had, since, for example, something like ChatGPT is just text, and the phrase "a picture is worth a thousand words" seems incredibly accurate, from my perspective. There's an incredible amount of non-text data out there still. Especially technical data…
Yes. One of the frontiers of current research seems to be multi-modal models.
Re: OpenAI’s CEO says the age of giant AI models is already over
#197I believe Altman, but the title is misleading.
Have we exhausted the value of larger models on current architecture? Probably yes. I trust OpenAI would throw more $ at it if there was anything left on the table.
Have we been here before? Also yes. I recall hearing similar things about LSTMs when they were in vogue.
Will the next game changing architecture require a huge model? Probably. Don’t see any sign these things are scaling _worse_ with more data/compute.
The age of huge models with current architecture could be over, but that started what, 5 years ago? Who cares?
Re: OpenAI’s CEO says the age of giant AI models is already over
#198Earlier quoted context omitted.
Good thing he got a bunch of companies to pony up the dough for LLM before he announced they where already over.
I don't think LLMs are over [0]. I think we're relatively close to a local optimum in terms of what can be achieved with current algorithms. But I think OpenAI is at least as likely as any other player to create the next paradigm, and that it's at least as likely as likely as any other player to develop the leading models within the next paradigm regardless of who actually publishes the research. Separately, I think…
It already is relatively trivial to fine-tune generative models for various use cases. Which implies huge gains to be had with targeted applications not just for niche players but also OpenAI and others to either build that fine-tuning into the base system, build ecosystems around it, or just purpose build applications on top.
Re: OpenAI’s CEO says the age of giant AI models is already over
#199Earlier quoted context omitted.
Does he even have any background in machine learning? I always found it bizarre that he was chosen to be OpenAI's CEO...
On the Lex Fridman podcast, he pretty much admitted he's not an AI (per se) and isn't the most excited about the tech (as he could be).
> he pretty much admitted he's not an AI
Yeah, I also had a hunch he wasn't an AI. (I assume you meant "AI researcher" there :))
All joking aside, I wonder how that's affecting company morale or their ability to attract top researchers. I know if I was a top AI researcher, I'd probably rather work at a company where the CEO was an expert in the field (all else being equal).
Re: OpenAI’s CEO says the age of giant AI models is already over
#200Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
https://twitter.com/SmokeAwayyy/status/1646670920214536193 Sam explicitly said that there won't be GPT-5 in the near future, which is pretty clear evidence unless he's blatantly lying in public speaking.
Well, "no GPT-5" isn't the same as saying "no new trained model", especially in the realm of marketing. Welcome to "GPT 2024" could be his next slogan.