401–410 of 525 posts
The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
dude someone lied their way into being president of the united states all while people fact checked him basically immediately after each lie. lying doesnt make a difference.
He’s not presenting false evidence here, he’s presenting a hunch. It’s a guess. No one is going to gain anything from this one way or another.
"don't bother competing with me. i have already won."
Earlier quoted context omitted.
or like a curve of model complexity versus results or whatever showing it asymptotically approaches whatever. actually there was a great paper from microsoft research from like 2001 on spam filtering where they demonstrated that model complexity necessary for spam filtering went down as the size of the data set went up. That paper, which i can't seem to find now, had a big impact on me as a researcher because it so c…
I believe this is the paper which you are referring to: https://aclanthology.org/P01-1005.pdf ("Scaling to Very Very Large Corpora for Natural Language Disambiguation" by Michele Banko and Eric Brill, Microsoft Research, 2001)
The point is that now we're at the point of diminishing return for increasing model size, unless we find a better modeling architecture than Transformer. I think this is likely true; while all the other companies underestimated the capability of transformer (including Google itself!), OpenAI made a fairly accurate bet on the transformer based on the scaling law, put all the efforts to squeeze it until the last drop a…
> (including Google itself!) Bet Google won’t make that mistake again, i.e. it won’t publish as much and will be much more careful about what it publishes, least they give a competitor a useful tool and get nothing in return - when the competitor (in this case very ironically named) goes full commercial and close source everything they can. Open collaboration in AI, at least when it comes to corporations, might have…
Interestingly what's makes ChatGPT work is the size of the model so I think they've found their dead stop.
Lot's of the bullish comments have been talking about how ChatGPT is a bit shit right now but will exponentially get better and I think the answer is now the progress will be much slower and more linear. That is if they can stay funded which is a very big if. As an org they are bleeding money.
Sounds sus
Sorry I know this adds little to the conversation
Earlier quoted context omitted.
Seems like a wild claim to make without any examples of gpt models which are bigger and no demonstrably better.
Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.
I'd rather have a LLM that thinks a bit longer than a LLM that spits out wrong answers immediately.
Sam Altman and OpenAI must be pretty nervous. They have first mover advantage but they hold no hook or moat. Unless they can somehow keep their improvements ahead of the rest of the industry then they'll be lost among a crowd.