I think Sam is referring to transition from "Deep" to "Long" learning [1]. What new emergent properties, if any, will 1 billion tokens unlock?
[1] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn...
OpenAI’s CEO says the age of giant AI models is already over
31–40 of 525 posts
Re: OpenAI’s CEO says the age of giant AI models is already over
#32Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
Right. They've already sucked in most of the good general sources of information. Adding vast amounts of low-quality content probably won't help much and might degrade the quality of the trained model.
Re: OpenAI’s CEO says the age of giant AI models is already over
#33Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
Video content (I don't know why someone flagged Jason for saying such, he is totally right)
Re: OpenAI’s CEO says the age of giant AI models is already over
#34The Age of Giants is over... The Age of Behemoths has begun! but sir, that means the same thing Throw this heretic into the pit of terror.
The pit of terror is full.
Fine, to the outhouse of madness then.
Before I get nuked from orbit for daring to entertain humor, if someone is running ahead of me in a marathon, and running so far ahead, yet still broadcasting things to the back for the slow people (like myself), then eventually we catch up to them, and they suddenly say, you know what guys, we should stop running in this direction, there's nothing to see here right before anyone else is able to verify the veracity of their statement, perhaps it would still be in the public interest for at least one person to verify what they are saying. Given how skeptical the internet at large has been of Musk's acquisition of a company, it's interesting that the skepticism is suddenly put on hold when looking at this part of his work...
Re: OpenAI’s CEO says the age of giant AI models is already over
#35An amusing thought I've had recently is whether LLMs are in the same league as the millions of monkeys at the keyboard, struggling to reproduce one of the complete works of William Shakespeare. But I think not, since monkeys probably don't "improve" noticeably with time or input.
But I think not, since monkeys probably don't "improve" noticeably with time or input.
Maybe once tons of bananas are introduced...
Re: OpenAI’s CEO says the age of giant AI models is already over
#36I'm no expert but doesn't the architecture of minigpt4 that's on the front page right now give some indication of what the future might look like?
eh, I haven't personally found a usecase for LLMs yet given the fact that you can't trust the output and it needs to be verified by a human (which might as well be just as time consuming/expensive as actually doing the task yourself)
I’d reconsider the “might as well just be as time consuming” thing. I see this argument about Copilot a lot, and it’s really wrong there, so it might be wrong here too.
Like, for most of the time I’m using it, Copilot saves me 30 seconds here and there and it takes me about a second to look at the line or two of code and go “yeah, that’s right”. It adds up, especially when I’m working with an unfamiliar language and forget which Collection type I’m going to need or something.
Re: OpenAI’s CEO says the age of giant AI models is already over
#37Once you've trained on the internet and most published books (and more...) what else is there to do? You can't scale up massively anymore.
Video content (I don't know why someone flagged Jason for saying such, he is totally right)
Looking at his post history, seems like he was shadowbanned.
Re: OpenAI’s CEO says the age of giant AI models is already over
#38The most comforting AI news I have read this year.
If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild
Machine learning is actually premised on being “simple” to implement. The more priors you hardcode with clever algorithms, the closer you get to what we already have. The point is to automate the process of learning. We do this now with relatively simple loss functions and models containing relatively simple parameters. The main stipulation is that they are all defined to be continuous so that you can use the chain rule from calculus to calculate the error with respect to every parameter without taking so long that it would never finish.
I agree that your suggested approach of applying cleverness to what we have now will probably produce better results. But that’s not going to stop better architectures, hardware and even entire regimes from being developed until we approach AGI.
My suspicion is that there’s still a few breakthroughs waiting to be made. I also suspect that sufficiently advanced models will make such breakthroughs easier to discover.
Re: OpenAI’s CEO says the age of giant AI models is already over
#39We arrived at the top of the tree in our journey to the moon.
No we haven't. the title is misleading. there's plenty of scale room left. part of it might just not be economical (parameter sie) but there's data. If you take this to mean, "we're at a dead end" you'd be very wrong
Re: OpenAI’s CEO says the age of giant AI models is already over
#40The most comforting AI news I have read this year.
If you are worried about AI, this shouldn't make you feel a ton better. GPT4 is just trained to predict the next word, a very simple but crude approach and look what it can do! Imagine when a dozen models are wired together and giving each other feedback with more clever training and algorithms on future faster hardware. It is still going to get wild
People think something magical happens when AI are wired together and give each other feedback.
Really you’re still just predicting the next word, but with extra steps.